Sep 10 15:40:16.345653 ip-10-0-138-241 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 10 15:40:16.345665 ip-10-0-138-241 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 10 15:40:16.345675 ip-10-0-138-241 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 10 15:40:16.346116 ip-10-0-138-241 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 10 15:40:26.562149 ip-10-0-138-241 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 10 15:40:26.562167 ip-10-0-138-241 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 78d72c88205e47f794923ac094f4d51c --
Sep 10 15:42:01.502724 ip-10-0-138-241 systemd[1]: Starting Kubernetes Kubelet...
Sep 10 15:42:01.907699 ip-10-0-138-241 kubenswrapper[2560]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.907699 ip-10-0-138-241 kubenswrapper[2560]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 10 15:42:01.907699 ip-10-0-138-241 kubenswrapper[2560]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.907699 ip-10-0-138-241 kubenswrapper[2560]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 10 15:42:01.907699 ip-10-0-138-241 kubenswrapper[2560]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 10 15:42:01.908922 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.908830    2560 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 10 15:42:01.914536 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914512    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.914536 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914534    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.914536 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914538    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914541    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914545    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914548    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914552    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914555    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914559    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914561    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914564    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914566    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914569    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914572    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914575    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914577    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914580    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914583    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914587    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914589    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914592    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914594    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.914649 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914597    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914600    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914602    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914605    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914607    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914610    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914613    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914616    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914618    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914621    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914623    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914625    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914629    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914632    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914635    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914638    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914642    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914646    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914649    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.915161 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914652    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914655    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914657    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914660    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914663    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914665    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914668    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914671    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914673    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914676    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914679    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914681    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914686    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914690    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914694    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914696    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914699    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914702    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914704    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914707    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.915903 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914709    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914711    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914715    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914719    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914721    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914724    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914727    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914730    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914733    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914736    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914738    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914741    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914744    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914746    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914749    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914751    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914754    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914758    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914761    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.916658 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914764    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914766    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914769    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914771    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914774    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914776    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914779    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914781    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914784    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914787    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914790    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914792    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914795    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914797    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914800    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914802    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914805    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914807    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.917433 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.914810    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917758    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917776    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917781    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917787    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917792    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917797    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917802    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917807    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917811    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917815    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917819    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917823    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917827    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917831    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917836    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917840    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917844    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917848    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917853    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.918193 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917857    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917861    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917865    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917870    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917874    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917878    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917882    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917886    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917890    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917894    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917899    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917903    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917907    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917912    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917916    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917921    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917925    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917929    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917933    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917938    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.919033 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917942    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917946    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917950    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917954    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917958    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917962    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917969    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917974    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917980    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917987    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.917993    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918001    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918006    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918012    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918017    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918022    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918027    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918031    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918035    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.919922 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918040    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918044    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918049    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918054    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918059    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918063    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918068    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918072    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918077    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918082    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918087    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918092    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918096    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918100    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918104    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918108    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918112    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918116    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918121    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918125    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.920770 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918130    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918134    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918138    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918145    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918149    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918156    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918160    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918164    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918169    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918173    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918180    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918184    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918188    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918192    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918198    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918203    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918208    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918213    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918218    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918223    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.921421 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918227    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918865    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918873    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918878    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918883    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918888    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918892    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918899    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918904    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918909    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918914    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918919    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918924    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918928    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918932    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918939    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918944    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918949    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918953    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918957    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.922024 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918961    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918965    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918969    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918973    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918978    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918982    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918986    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918990    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918994    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.918999    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919003    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919007    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919011    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919016    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919020    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919024    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919028    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919032    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919036    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919040    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.922561 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919044    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919048    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919051    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919055    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919059    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919065    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919069    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919073    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919077    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919081    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919088    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919093    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919097    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919101    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919105    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919109    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919114    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919118    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919122    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.923154 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919126    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919130    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919135    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919139    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919143    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919147    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919151    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919155    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919159    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919164    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919168    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919172    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919176    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919180    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919184    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919188    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919192    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919196    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919200    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919204    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.923787 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919208    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919212    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919217    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919221    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919230    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919235    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919239    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919243    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919247    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919252    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919256    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919260    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919264    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919268    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919273    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919277    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919281    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919285    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919292    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.924273 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919297    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.919302    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919423    2560 flags.go:64] FLAG: --address="0.0.0.0"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919481    2560 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919491    2560 flags.go:64] FLAG: --anonymous-auth="true"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919498    2560 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919505    2560 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919510    2560 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919517    2560 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919524    2560 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919529    2560 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919534    2560 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919539    2560 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919544    2560 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919549    2560 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919554    2560 flags.go:64] FLAG: --cgroup-root=""
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919559    2560 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919563    2560 flags.go:64] FLAG: --client-ca-file=""
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919569    2560 flags.go:64] FLAG: --cloud-provider="external"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919574    2560 flags.go:64] FLAG: --cluster-dns="[]"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919581    2560 flags.go:64] FLAG: --cluster-domain=""
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919586    2560 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919591    2560 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919596    2560 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 10 15:42:01.924905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919601    2560 flags.go:64] FLAG: --container-log-max-files="5"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919607    2560 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919612    2560 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919617    2560 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919623    2560 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919627    2560 flags.go:64] FLAG: --contention-profiling="false"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919632    2560 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919636    2560 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919641    2560 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919646    2560 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919652    2560 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919656    2560 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919661    2560 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919665    2560 flags.go:64] FLAG: --enable-load-reader="false"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919670    2560 flags.go:64] FLAG: --enable-server="true"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919675    2560 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919682    2560 flags.go:64] FLAG: --event-burst="100"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919687    2560 flags.go:64] FLAG: --event-qps="50"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919692    2560 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919696    2560 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919701    2560 flags.go:64] FLAG: --eviction-hard=""
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919708    2560 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919712    2560 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919717    2560 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919722    2560 flags.go:64] FLAG: --eviction-soft=""
Sep 10 15:42:01.925589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919726    2560 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919731    2560 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919736    2560 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919742    2560 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919746    2560 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919751    2560 flags.go:64] FLAG: --fail-swap-on="true"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919755    2560 flags.go:64] FLAG: --feature-gates=""
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919762    2560 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919767    2560 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919772    2560 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919777    2560 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919782    2560 flags.go:64] FLAG: --healthz-port="10248"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919787    2560 flags.go:64] FLAG: --help="false"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919792    2560 flags.go:64] FLAG: --hostname-override="ip-10-0-138-241.ec2.internal"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919797    2560 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919801    2560 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919806    2560 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919811    2560 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919817    2560 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919824    2560 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919829    2560 flags.go:64] FLAG: --image-service-endpoint=""
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919834    2560 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919838    2560 flags.go:64] FLAG: --kube-api-burst="100"
Sep 10 15:42:01.926213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919843    2560 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919850    2560 flags.go:64] FLAG: --kube-api-qps="50"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919855    2560 flags.go:64] FLAG: --kube-reserved=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919859    2560 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919864    2560 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919869    2560 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919873    2560 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919878    2560 flags.go:64] FLAG: --lock-file=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919882    2560 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919887    2560 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919892    2560 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919901    2560 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919908    2560 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919912    2560 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919918    2560 flags.go:64] FLAG: --logging-format="text"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919922    2560 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919927    2560 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919933    2560 flags.go:64] FLAG: --manifest-url=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919937    2560 flags.go:64] FLAG: --manifest-url-header=""
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919944    2560 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919948    2560 flags.go:64] FLAG: --max-open-files="1000000"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919955    2560 flags.go:64] FLAG: --max-pods="110"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919960    2560 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919965    2560 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919969    2560 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 10 15:42:01.926820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919974    2560 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919979    2560 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919983    2560 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.919990    2560 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920002    2560 flags.go:64] FLAG: --node-status-max-images="50"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920007    2560 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920012    2560 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920017    2560 flags.go:64] FLAG: --pod-cidr=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920021    2560 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:8ccbacc999dbf9abad8f4009240188312b683f36ced4763eed166c1500c164b1"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920031    2560 flags.go:64] FLAG: --pod-manifest-path=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920035    2560 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920040    2560 flags.go:64] FLAG: --pods-per-core="0"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920045    2560 flags.go:64] FLAG: --port="10250"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920050    2560 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920055    2560 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-0a74a79df690bb6f3"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920060    2560 flags.go:64] FLAG: --qos-reserved=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920065    2560 flags.go:64] FLAG: --read-only-port="10255"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920070    2560 flags.go:64] FLAG: --register-node="true"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920074    2560 flags.go:64] FLAG: --register-with-taints=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920080    2560 flags.go:64] FLAG: --registry-burst="10"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920095    2560 flags.go:64] FLAG: --registry-qps="5"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920101    2560 flags.go:64] FLAG: --reserved-cpus=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920106    2560 flags.go:64] FLAG: --reserved-memory=""
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920112    2560 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920117    2560 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 10 15:42:01.927448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920122    2560 flags.go:64] FLAG: --rotate-certificates="false"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920127    2560 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920131    2560 flags.go:64] FLAG: --runonce="false"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920136    2560 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920141    2560 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920146    2560 flags.go:64] FLAG: --seccomp-default="false"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920151    2560 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920155    2560 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920160    2560 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920164    2560 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920171    2560 flags.go:64] FLAG: --storage-driver-password="root"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920176    2560 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920181    2560 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920185    2560 flags.go:64] FLAG: --storage-driver-user="root"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920190    2560 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920195    2560 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920199    2560 flags.go:64] FLAG: --system-cgroups=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920204    2560 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920213    2560 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920217    2560 flags.go:64] FLAG: --tls-cert-file=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920222    2560 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920235    2560 flags.go:64] FLAG: --tls-min-version=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920239    2560 flags.go:64] FLAG: --tls-private-key-file=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920244    2560 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920249    2560 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 10 15:42:01.928142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920253    2560 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920258    2560 flags.go:64] FLAG: --v="2"
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920264    2560 flags.go:64] FLAG: --version="false"
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920279    2560 flags.go:64] FLAG: --vmodule=""
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920290    2560 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.920295    2560 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920575    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920583    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920588    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920594    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920598    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920603    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920607    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920611    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920615    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920619    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920623    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920631    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920636    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920640    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920644    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.928888 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920647    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920652    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920655    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920660    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920664    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920668    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920672    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920676    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920680    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920684    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920688    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920692    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920696    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920700    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920704    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920721    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920726    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920732    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920738    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.929419 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920743    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920747    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920751    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920756    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920759    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920764    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920768    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920772    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920776    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920782    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920786    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920790    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920793    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920797    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920802    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920805    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920810    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920815    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920819    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920823    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920827    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.929900 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920831    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920835    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920839    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920843    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920847    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920851    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920855    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920861    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920876    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920882    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920888    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920894    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920899    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920904    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920909    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920914    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920919    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920923    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920927    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.930443 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920931    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920937    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920941    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920946    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920950    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920954    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920958    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920962    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920966    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920970    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920974    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920978    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920983    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920987    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920991    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920995    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.920999    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921003    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921007    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921011    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.930904 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921018    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921022    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921034    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921039    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.921043    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.921693    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.929418    2560 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.929437    2560 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.929456    2560 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.929461    2560 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929529    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929535    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929538    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929541    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929544    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.931408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929547    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929549    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929552    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929555    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929557    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929560    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929562    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929565    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929568    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929570    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929572    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929575    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929577    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929580    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929582    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929585    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929588    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929591    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929593    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929597    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.931794 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929599    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929602    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929604    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929607    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929609    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929614    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929618    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929622    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929625    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929627    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929630    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929632    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929635    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929637    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929640    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929643    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929646    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929648    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929650    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929653    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.932323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929655    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929659    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929663    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929665    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929668    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929670    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929672    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929675    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929678    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929681    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929683    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929686    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929689    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929691    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929694    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929696    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929700    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929702    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929705    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.932898 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929707    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929710    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929712    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929715    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929718    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929720    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929722    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929725    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929727    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929730    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929732    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929735    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929737    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929747    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929751    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929755    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929758    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929761    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929764    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.933367 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929767    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929770    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929773    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929776    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929778    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929781    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929784    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929787    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929790    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929793    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929796    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929798    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929801    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929803    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929806    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.933833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929808    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.929814    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929932    2560 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929937    2560 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929940    2560 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929942    2560 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929946    2560 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929949    2560 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929951    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929954    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929956    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929959    2560 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929961    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929964    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929966    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 10 15:42:01.934199 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929969    2560 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929971    2560 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929973    2560 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929976    2560 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929979    2560 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929981    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929984    2560 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929986    2560 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929989    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929991    2560 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.929994    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930001    2560 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930004    2560 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930006    2560 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930009    2560 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930012    2560 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930014    2560 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930016    2560 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930019    2560 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 10 15:42:01.934593 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930021    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930024    2560 feature_gate.go:328] unrecognized feature gate: Example
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930026    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930029    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930032    2560 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930034    2560 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930036    2560 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930039    2560 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930041    2560 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930044    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930046    2560 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930049    2560 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930051    2560 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930053    2560 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930056    2560 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930058    2560 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930060    2560 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930063    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930065    2560 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 10 15:42:01.935063 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930068    2560 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930070    2560 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930073    2560 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930075    2560 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930077    2560 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930081    2560 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930083    2560 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930086    2560 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930089    2560 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930092    2560 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930096    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930099    2560 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930101    2560 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930105    2560 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930109    2560 feature_gate.go:328] unrecognized feature gate: Example2
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930112    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930115    2560 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930118    2560 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930120    2560 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930123    2560 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 10 15:42:01.935547 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930126    2560 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930129    2560 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930131    2560 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930134    2560 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930136    2560 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930139    2560 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930141    2560 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930143    2560 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930146    2560 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930148    2560 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930151    2560 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930153    2560 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930156    2560 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930158    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930161    2560 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930163    2560 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930166    2560 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930168    2560 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930171    2560 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930174    2560 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 10 15:42:01.936038 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930176    2560 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930179    2560 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930182    2560 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930184    2560 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930188    2560 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930191    2560 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930194    2560 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:01.930197    2560 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.930202    2560 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.930332    2560 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.933105    2560 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.933966    2560 server.go:1019] "Starting client certificate rotation"
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.934067    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:01.936610 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.934098    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 10 15:42:01.936933 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.936707    2560 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:01.940119 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.940096    2560 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 10 15:42:01.951669 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.951638    2560 log.go:25] "Validated CRI v1 runtime API"
Sep 10 15:42:01.955809 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.955788    2560 log.go:25] "Validated CRI v1 image API"
Sep 10 15:42:01.956757 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.956735    2560 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 10 15:42:01.960863 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.960829    2560 fs.go:135] Filesystem UUIDs: map[242b84e3-bdd5-4a41-9de5-4a2bd0cfe25a:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2 84f5336d-5891-4c26-8d54-b938293510fa:/dev/nvme0n1p4]
Sep 10 15:42:01.960955 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.960859    2560 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 10 15:42:01.966280 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.966251    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:01.966512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.966365    2560 manager.go:216] Machine: {Timestamp:2026-09-10 15:42:01.964689261 +0000 UTC m=+0.344636317 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099591 MemoryCapacity:32812158976 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec273b9ec82930603a24e6520262bf9c SystemUUID:ec273b9e-c829-3060-3a24-e6520262bf9c BootID:78d72c88-205e-47f7-9492-3ac094f4d51c Filesystems:[{Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16406081536 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16406077440 Type:vfs Inodes:4005390 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6562435072 Type:vfs Inodes:819200 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:c6:44:9d:60:6d Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:c6:44:9d:60:6d Speed:0 Mtu:9001} {Name:ovs-system MacAddress:4e:50:ae:cd:aa:f5 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812158976 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 10 15:42:01.966512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.966509    2560 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 10 15:42:01.966664 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.966647    2560 manager.go:232] Version: {KernelVersion:5.14.0-570.138.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260901-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 10 15:42:01.970296 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.970243    2560 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 10 15:42:01.970514 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.970298    2560 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-138-241.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 10 15:42:01.971156 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.971144    2560 topology_manager.go:138] "Creating topology manager with none policy"
Sep 10 15:42:01.971208 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.971168    2560 container_manager_linux.go:309] "Creating device plugin manager"
Sep 10 15:42:01.971208 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.971181    2560 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:01.971978 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.971962    2560 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 10 15:42:01.972824 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.972809    2560 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 10 15:42:01.975942 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.975923    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:01.976097 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976086    2560 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 10 15:42:01.976167 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976132    2560 kubelet.go:486] "Attempting to sync node with API server"
Sep 10 15:42:01.976167 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976155    2560 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 10 15:42:01.976253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976178    2560 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 10 15:42:01.976253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976193    2560 kubelet.go:398] "Adding apiserver pod source"
Sep 10 15:42:01.976253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976207    2560 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 10 15:42:01.976726 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976711    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:01.976791 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.976734    2560 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 10 15:42:01.979322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.979304    2560 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 10 15:42:01.981580 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.981564    2560 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 10 15:42:01.981631 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.981598    2560 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 10 15:42:01.982910 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982897    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982914    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982923    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982932    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982942    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982948    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982961    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 10 15:42:01.982966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982966    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 10 15:42:01.983168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982973    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 10 15:42:01.983168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982978    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 10 15:42:01.983168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982988    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 10 15:42:01.983168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.982998    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 10 15:42:01.984373 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.984362    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 10 15:42:01.984373 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.984374    2560 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 10 15:42:01.988753 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.988730    2560 server.go:1268] "Started kubelet"
Sep 10 15:42:01.988856 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:01.988800    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 10 15:42:01.988856 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:01.988803    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-138-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 10 15:42:01.988953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.988899    2560 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 10 15:42:01.988998 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.988970    2560 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-138-241.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 10 15:42:01.988998 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.988980    2560 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 10 15:42:01.989082 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.989008    2560 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 10 15:42:01.989648 ip-10-0-138-241 systemd[1]: Started Kubernetes Kubelet.
Sep 10 15:42:01.990635 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.990552    2560 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 10 15:42:01.994582 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.994558    2560 server.go:310] "Adding debug handlers to kubelet server"
Sep 10 15:42:01.995753 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:01.995722    2560 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 10 15:42:01.998219 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:01.997392    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-241.ec2.internal.18d3fff904ccd244  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-241.ec2.internal,UID:ip-10-0-138-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-138-241.ec2.internal,},FirstTimestamp:2026-09-10 15:42:01.9886905 +0000 UTC m=+0.368637562,LastTimestamp:2026-09-10 15:42:01.9886905 +0000 UTC m=+0.368637562,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-241.ec2.internal,}"
Sep 10 15:42:01.999450 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.999431    2560 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 10 15:42:01.999450 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:01.999442    2560 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:01.999598 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:01.999538    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.000544 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.000501    2560 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 10 15:42:02.000737 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.000704    2560 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 10 15:42:02.000737 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.000730    2560 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 10 15:42:02.000873 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.000856    2560 reconstruct.go:97] "Volume reconstruction finished"
Sep 10 15:42:02.000873 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.000867    2560 reconciler.go:29] "Reconciler: start to sync state"
Sep 10 15:42:02.002488 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.002409    2560 factory.go:55] Registering systemd factory
Sep 10 15:42:02.002488 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.002428    2560 factory.go:223] Registration of the systemd container factory successfully
Sep 10 15:42:02.002977 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.002962    2560 factory.go:153] Registering CRI-O factory
Sep 10 15:42:02.003040 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.002981    2560 factory.go:223] Registration of the crio container factory successfully
Sep 10 15:42:02.003040 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.003034    2560 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 10 15:42:02.003120 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.003051    2560 factory.go:103] Registering Raw factory
Sep 10 15:42:02.003120 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.003066    2560 manager.go:1196] Started watching for new ooms in manager
Sep 10 15:42:02.003704 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.003677    2560 manager.go:349] Starting recovery of all containers
Sep 10 15:42:02.009059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.008897    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 10 15:42:02.010435 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.010396    2560 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-138-241.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 10 15:42:02.010534 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.010518    2560 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 10 15:42:02.013714 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.013698    2560 manager.go:354] Recovery completed
Sep 10 15:42:02.015186 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.015166    2560 watcher.go:152] Failed to watch directory "/sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service": inotify_add_watch /sys/fs/cgroup/system.slice/systemd-update-utmp-runlevel.service: no such file or directory
Sep 10 15:42:02.018206 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.018191    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.020262 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020245    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.020411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020275    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.020411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020285    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.020902 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020889    2560 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 10 15:42:02.020957 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020902    2560 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 10 15:42:02.020957 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020920    2560 state_mem.go:36] "Initialized new in-memory state store"
Sep 10 15:42:02.021010 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.020896    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.021712 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.021701    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.021771 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.021727    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.021771 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.021750    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.022218 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.022196    2560 policy_none.go:49] "None policy: Start"
Sep 10 15:42:02.022218 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.022211    2560 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 10 15:42:02.022218 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.022221    2560 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 10 15:42:02.022412 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.022328    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-241.ec2.internal.18d3fff906ae907a  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-241.ec2.internal,UID:ip-10-0-138-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-138-241.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-138-241.ec2.internal,},FirstTimestamp:2026-09-10 15:42:02.02026201 +0000 UTC m=+0.400209066,LastTimestamp:2026-09-10 15:42:02.02026201 +0000 UTC m=+0.400209066,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-241.ec2.internal,}"
Sep 10 15:42:02.023440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.023427    2560 policy_none.go:47] "Start"
Sep 10 15:42:02.028569 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.028546    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22v4f"
Sep 10 15:42:02.030368 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.030279    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-241.ec2.internal.18d3fff906aed729  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-241.ec2.internal,UID:ip-10-0-138-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-138-241.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-138-241.ec2.internal,},FirstTimestamp:2026-09-10 15:42:02.020280105 +0000 UTC m=+0.400227161,LastTimestamp:2026-09-10 15:42:02.020280105 +0000 UTC m=+0.400227161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-241.ec2.internal,}"
Sep 10 15:42:02.032746 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.032672    2560 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-138-241.ec2.internal.18d3fff906aef88e  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-138-241.ec2.internal,UID:ip-10-0-138-241.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-138-241.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-138-241.ec2.internal,},FirstTimestamp:2026-09-10 15:42:02.020288654 +0000 UTC m=+0.400235710,LastTimestamp:2026-09-10 15:42:02.020288654 +0000 UTC m=+0.400235710,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-138-241.ec2.internal,}"
Sep 10 15:42:02.036230 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.036209    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-22v4f"
Sep 10 15:42:02.064073 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064045    2560 manager.go:337] "Starting Device Plugin manager"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.064109    2560 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064123    2560 server.go:87] "Starting device plugin registration server"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064494    2560 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064505    2560 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064606    2560 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064695    2560 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.064704    2560 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.065275    2560 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 10 15:42:02.072469 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.065320    2560 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.109656 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.109627    2560 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 10 15:42:02.109821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.109688    2560 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 10 15:42:02.109821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.109718    2560 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 10 15:42:02.109821 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.109781    2560 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 10 15:42:02.112458 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.112426    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:02.165501 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.165386    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.166505 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.166488    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.166583 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.166519    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.166583 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.166529    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.166583 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.166551    2560 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.174947 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.174926    2560 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.174999 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.174959    2560 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-138-241.ec2.internal\": node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.189085 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.189054    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.210427 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.210379    2560 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"]
Sep 10 15:42:02.210496 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.210473    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.212139 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.212119    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.212243 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.212155    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.212243 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.212171    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.213522 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.213508    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.213636 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.213620    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.213786 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.213655    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.214361 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214330    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.214439 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214376    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.214439 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214386    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.214439 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214333    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.214534 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214443    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.214534 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.214456    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.215704 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.215680    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.215818 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.215711    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:02.216498 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.216483    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:02.216593 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.216509    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:02.216593 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.216519    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:02.245392 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.245362    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-241.ec2.internal\" not found" node="ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.249143 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.249120    2560 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-138-241.ec2.internal\" not found" node="ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.289526 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.289485    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.303071 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.303043    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ace8c758eb0a5cfdc6b4f1e8d25f1a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-241.ec2.internal\" (UID: \"17ace8c758eb0a5cfdc6b4f1e8d25f1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.303152 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.303073    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.303152 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.303089    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.390241 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.390209    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.403657 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ace8c758eb0a5cfdc6b4f1e8d25f1a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-241.ec2.internal\" (UID: \"17ace8c758eb0a5cfdc6b4f1e8d25f1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.403730 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403660    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.403730 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403678    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.403792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/17ace8c758eb0a5cfdc6b4f1e8d25f1a-config\") pod \"kube-apiserver-proxy-ip-10-0-138-241.ec2.internal\" (UID: \"17ace8c758eb0a5cfdc6b4f1e8d25f1a\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.403820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403759    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.403820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.403756    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/a52e03e9a1cd697f3e8b90f6f1d339ef-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal\" (UID: \"a52e03e9a1cd697f3e8b90f6f1d339ef\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.491146 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.491062    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.547630 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.547593    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.552822 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.552797    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:02.591836 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.591804    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.692466 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.692414    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.793008 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.792906    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.893568 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.893534    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:02.934197 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.934157    2560 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 10 15:42:02.934675 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:02.934330    2560 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 10 15:42:02.994465 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:02.994432    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:03.040410 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.040314    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-09 15:37:01 +0000 UTC" deadline="2028-04-30 09:10:22.318244618 +0000 UTC"
Sep 10 15:42:03.040410 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.040398    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="14345h28m19.277852218s"
Sep 10 15:42:03.088754 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.088677    2560 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 10 15:42:03.090512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.090489    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientMemory"
Sep 10 15:42:03.090651 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.090525    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasNoDiskPressure"
Sep 10 15:42:03.090651 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.090545    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeHasSufficientPID"
Sep 10 15:42:03.094615 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:03.094589    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:03.127723 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:03.127685    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda52e03e9a1cd697f3e8b90f6f1d339ef.slice/crio-28c36386c4377fb5a1aafdc9bf49fecbd2405f22af988a59404a862338cd5d0e WatchSource:0}: Error finding container 28c36386c4377fb5a1aafdc9bf49fecbd2405f22af988a59404a862338cd5d0e: Status 404 returned error can't find the container with id 28c36386c4377fb5a1aafdc9bf49fecbd2405f22af988a59404a862338cd5d0e
Sep 10 15:42:03.127995 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:03.127972    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17ace8c758eb0a5cfdc6b4f1e8d25f1a.slice/crio-d7e5432edc1798e281f7a5f90769dbef0de79386ff14c58021d6048501725087 WatchSource:0}: Error finding container d7e5432edc1798e281f7a5f90769dbef0de79386ff14c58021d6048501725087: Status 404 returned error can't find the container with id d7e5432edc1798e281f7a5f90769dbef0de79386ff14c58021d6048501725087
Sep 10 15:42:03.134409 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.134382    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:42:03.195308 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:03.195270    2560 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-138-241.ec2.internal\" not found"
Sep 10 15:42:03.248641 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.248611    2560 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:03.287453 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.287423    2560 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:03.300276 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.300246    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:03.311171 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.311147    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:03.312138 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.312124    2560 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal"
Sep 10 15:42:03.319351 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.319322    2560 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 10 15:42:03.366887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.366816    2560 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:03.976612 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.976574    2560 apiserver.go:52] "Watching apiserver"
Sep 10 15:42:03.986261 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.986208    2560 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 10 15:42:03.987002 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.986961    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn","openshift-cluster-node-tuning-operator/tuned-p2x9j","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal","openshift-multus/multus-z5cwm","openshift-network-diagnostics/network-check-target-msr9m","openshift-network-operator/iptables-alerter-2kmvk","kube-system/global-pull-secret-syncer-drwxg","kube-system/konnectivity-agent-zjv5t","openshift-image-registry/node-ca-ctnsr","openshift-multus/multus-additional-cni-plugins-8fqqm","openshift-multus/network-metrics-daemon-bs58t","openshift-ovn-kubernetes/ovnkube-node-snvlb"]
Sep 10 15:42:03.988624 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.988597    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:03.988748 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:03.988680    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:03.989729 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.989690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:03.990827 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.990796    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:03.992165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.992147    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.992165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.992160    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.992390 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.992361    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:03.993065 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993049    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.993236 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993224    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 10 15:42:03.993405 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993389    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 10 15:42:03.993503 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993463    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-wm8km\""
Sep 10 15:42:03.993503 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993479    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-vsspx\""
Sep 10 15:42:03.993596 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993566    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.993683 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.993667    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:03.994829 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.994765    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 10 15:42:03.995307 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.995290    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:03.996015 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.995837    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.996015 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.995863    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.996015 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.995874    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 10 15:42:03.996015 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.995903    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-nj7mv\""
Sep 10 15:42:03.996238 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.996060    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.996400 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.996382    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.996468 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.996416    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-f2s6q\""
Sep 10 15:42:03.996991 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.996972    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:03.997067 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:03.997044    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:03.997144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.997129    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 10 15:42:03.997471 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.997456    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-5tfsg\""
Sep 10 15:42:03.998398 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.998031    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 10 15:42:03.998398 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.998232    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 10 15:42:03.999154 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.999130    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:03.999881 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:03.999863    2560 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 10 15:42:04.000673 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.000655    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.001376 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.001359    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 10 15:42:04.001505 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.001490    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 10 15:42:04.001604 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.001589    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-mr4ll\""
Sep 10 15:42:04.001692 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.001674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.001780 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.001739    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:04.002821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.002744    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 10 15:42:04.002821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.002774    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 10 15:42:04.002968 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.002748    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-lj8xg\""
Sep 10 15:42:04.003296 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.003282    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.005352 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.005305    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 10 15:42:04.005440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.005399    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 10 15:42:04.005564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.005548    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-s29k5\""
Sep 10 15:42:04.006385 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.006368    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 10 15:42:04.007224 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.006630    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 10 15:42:04.007224 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.006751    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 10 15:42:04.007224 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.006889    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 10 15:42:04.012540 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-socket-dir-parent\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012552    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-bin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012569    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-tuned\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012584    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-registration-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012605    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-device-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012625    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-multus\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.012652 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sxtzn\" (UniqueName: \"kubernetes.io/projected/6c95eec8-944e-4836-b0c4-8601846bb4e5-kube-api-access-sxtzn\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012665    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-socket-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nzsfl\" (UniqueName: \"kubernetes.io/projected/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kube-api-access-nzsfl\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012771    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79fa45c1-bfe8-4cfb-b02b-18381665958b-host\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012791    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-modprobe-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-conf-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012856    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cnibin\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012879    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.012913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012900    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sszc9\" (UniqueName: \"kubernetes.io/projected/1de0b0c9-d69d-498a-9e30-37eb11daaa20-kube-api-access-sszc9\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012920    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-sys-fs\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012949    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-lib-modules\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.012998    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013026    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-conf\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013082    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-cni-binary-copy\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013096    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-kubelet\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013111    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-daemon-config\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013124    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-kubernetes\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013150    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-sys\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013173    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-hostroot\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013668 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013579    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013668 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013599    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-multus-certs\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013668 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysconfig\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013668 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.013668 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013647    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7v2mn\" (UniqueName: \"kubernetes.io/projected/85f5d22b-9e6d-4b28-b142-871a195b0cbf-kube-api-access-7v2mn\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.013792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013680    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.013792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-konnectivity-ca\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.013792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013738    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jj72d\" (UniqueName: \"kubernetes.io/projected/79fa45c1-bfe8-4cfb-b02b-18381665958b-kube-api-access-jj72d\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.013872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-k8s-cni-cncf-io\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.013872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013821    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-var-lib-kubelet\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013843    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5tvd2\" (UniqueName: \"kubernetes.io/projected/5e7b1b86-b666-4320-acf4-b0193d8eb053-kube-api-access-5tvd2\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.013872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.013983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013886    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-kubelet-config\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.013983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013917    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.013983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rkll5\" (UniqueName: \"kubernetes.io/projected/5ad42386-cffd-4d60-b3e8-f1ba059b1608-kube-api-access-rkll5\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.013981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-system-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-netns\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014042    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-host\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014060    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-tmp\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014090    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-system-cni-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.014117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85f5d22b-9e6d-4b28-b142-871a195b0cbf-iptables-alerter-script\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014132    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-cnibin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-etc-kubernetes\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014160    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85f5d22b-9e6d-4b28-b142-871a195b0cbf-host-slash\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014196    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-agent-certs\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014235    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79fa45c1-bfe8-4cfb-b02b-18381665958b-serviceca\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014260    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014282    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-systemd\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014296    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-dbus\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.014363 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-run\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.014646 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.014379    2560 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 10 15:42:04.038819 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.038792    2560 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn84k"
Sep 10 15:42:04.049059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.049038    2560 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-fn84k"
Sep 10 15:42:04.102083 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.102049    2560 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 10 15:42:04.114456 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114400    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal" event={"ID":"17ace8c758eb0a5cfdc6b4f1e8d25f1a","Type":"ContainerStarted","Data":"d7e5432edc1798e281f7a5f90769dbef0de79386ff14c58021d6048501725087"}
Sep 10 15:42:04.114456 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79fa45c1-bfe8-4cfb-b02b-18381665958b-host\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114480    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-modprobe-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114521    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-conf-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114542    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cnibin\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114552    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/79fa45c1-bfe8-4cfb-b02b-18381665958b-host\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114563    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114588    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sszc9\" (UniqueName: \"kubernetes.io/projected/1de0b0c9-d69d-498a-9e30-37eb11daaa20-kube-api-access-sszc9\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114609    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-sys-fs\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114635    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.114672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114656    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-lib-modules\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114672    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-modprobe-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114677    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114721    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114746    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-systemd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114762    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-config\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114777    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-conf\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114791    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-cni-binary-copy\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114809    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-kubelet\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-daemon-config\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114844    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-tuning-conf-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114848    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-kubernetes\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-sys\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114905    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-kubernetes\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114919    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-netd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114941    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-hostroot\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114961    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114967    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-etc-selinux\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.114981    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-log-socket\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115002    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-multus-certs\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115024    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysconfig\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115059    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-conf\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115077    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7v2mn\" (UniqueName: \"kubernetes.io/projected/85f5d22b-9e6d-4b28-b142-871a195b0cbf-kube-api-access-7v2mn\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115121    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-conf-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115122    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-konnectivity-ca\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115158    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jj72d\" (UniqueName: \"kubernetes.io/projected/79fa45c1-bfe8-4cfb-b02b-18381665958b-kube-api-access-jj72d\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115163    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cnibin\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115182    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-k8s-cni-cncf-io\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115203    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-var-lib-kubelet\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115222    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5tvd2\" (UniqueName: \"kubernetes.io/projected/5e7b1b86-b666-4320-acf4-b0193d8eb053-kube-api-access-5tvd2\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115245    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115278    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-kubelet-config\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.115595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115318    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75d94286-3829-46c1-8f14-3766f5477caf-ovn-node-metrics-cert\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115369    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rkll5\" (UniqueName: \"kubernetes.io/projected/5ad42386-cffd-4d60-b3e8-f1ba059b1608-kube-api-access-rkll5\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-system-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115453    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-netns\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115476    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-host\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-tmp\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115522    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-system-cni-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115547    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85f5d22b-9e6d-4b28-b142-871a195b0cbf-iptables-alerter-script\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115572    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-ovn\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115605    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-cni-binary-copy\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-cnibin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115640    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-kubelet\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115662    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-kubelet\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-bin\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115689    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-binary-copy\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115694    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-script-lib\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115868    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-etc-kubernetes\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85f5d22b-9e6d-4b28-b142-871a195b0cbf-host-slash\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-agent-certs\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115943    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-slash\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115966    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-node-log\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.115990    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pcbss\" (UniqueName: \"kubernetes.io/projected/75d94286-3829-46c1-8f14-3766f5477caf-kube-api-access-pcbss\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116014    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79fa45c1-bfe8-4cfb-b02b-18381665958b-serviceca\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116037    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116059    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-systemd\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116082    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-dbus\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116106    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-systemd-units\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116128    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-netns\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116152    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116177    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-run\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116199    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-var-lib-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116222    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-env-overrides\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-daemon-config\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.116830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116248    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/85f5d22b-9e6d-4b28-b142-871a195b0cbf-iptables-alerter-script\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-socket-dir-parent\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-bin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-socket-dir-parent\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-tuned\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116368    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-registration-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-sys-fs\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116422    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-registration-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116447    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-device-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116474    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-etc-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116514    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-cnibin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116516    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-lib-modules\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116556    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-multus\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116611    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sxtzn\" (UniqueName: \"kubernetes.io/projected/6c95eec8-944e-4836-b0c4-8601846bb4e5-kube-api-access-sxtzn\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116650    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.117592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116661    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-socket-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nzsfl\" (UniqueName: \"kubernetes.io/projected/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kube-api-access-nzsfl\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.116718    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.116792    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.616762346 +0000 UTC m=+2.996709422 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116890    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-sys\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.116967    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-device-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117002    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-bin\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117040    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-var-lib-cni-multus\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117041    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-system-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117089    2560 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117124    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-hostroot\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117161    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-netns\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.117192    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117201    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-host\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kubelet-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117327    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/4cbf0868-ca1b-4c92-9232-9e08f6577e36-socket-dir\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117413    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysctl-d\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118419 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.117544    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.61752224 +0000 UTC m=+2.997469297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117635    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/79fa45c1-bfe8-4cfb-b02b-18381665958b-serviceca\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117696    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-multus-cni-dir\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-systemd\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117757    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/1de0b0c9-d69d-498a-9e30-37eb11daaa20-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-dbus\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117801    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-multus-certs\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117842    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-sysconfig\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117852    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-run\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117882    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-kubelet-config\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117912    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-etc-kubernetes\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117945    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/85f5d22b-9e6d-4b28-b142-871a195b0cbf-host-slash\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117956    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-konnectivity-ca\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.117976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal" event={"ID":"a52e03e9a1cd697f3e8b90f6f1d339ef","Type":"ContainerStarted","Data":"28c36386c4377fb5a1aafdc9bf49fecbd2405f22af988a59404a862338cd5d0e"}
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.118057    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/5e7b1b86-b666-4320-acf4-b0193d8eb053-var-lib-kubelet\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.118152    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/1de0b0c9-d69d-498a-9e30-37eb11daaa20-system-cni-dir\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.118994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.118207    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/6c95eec8-944e-4836-b0c4-8601846bb4e5-host-run-k8s-cni-cncf-io\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.122362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.121081    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-etc-tuned\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.122362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.121100    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/5e7b1b86-b666-4320-acf4-b0193d8eb053-tmp\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.122362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.121385    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752-agent-certs\") pod \"konnectivity-agent-zjv5t\" (UID: \"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752\") " pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.127896 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.127861    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:04.127896 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.127887    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:04.127896 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.127898    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.128110 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.127950    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:04.627937298 +0000 UTC m=+3.007884362 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.134310 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.134231    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5tvd2\" (UniqueName: \"kubernetes.io/projected/5e7b1b86-b666-4320-acf4-b0193d8eb053-kube-api-access-5tvd2\") pod \"tuned-p2x9j\" (UID: \"5e7b1b86-b666-4320-acf4-b0193d8eb053\") " pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.134458 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.134397    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jj72d\" (UniqueName: \"kubernetes.io/projected/79fa45c1-bfe8-4cfb-b02b-18381665958b-kube-api-access-jj72d\") pod \"node-ca-ctnsr\" (UID: \"79fa45c1-bfe8-4cfb-b02b-18381665958b\") " pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.134516 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.134487    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nzsfl\" (UniqueName: \"kubernetes.io/projected/4cbf0868-ca1b-4c92-9232-9e08f6577e36-kube-api-access-nzsfl\") pod \"aws-ebs-csi-driver-node-zp5pn\" (UID: \"4cbf0868-ca1b-4c92-9232-9e08f6577e36\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.134563 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.134547    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sxtzn\" (UniqueName: \"kubernetes.io/projected/6c95eec8-944e-4836-b0c4-8601846bb4e5-kube-api-access-sxtzn\") pod \"multus-z5cwm\" (UID: \"6c95eec8-944e-4836-b0c4-8601846bb4e5\") " pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.134890 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.134873    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sszc9\" (UniqueName: \"kubernetes.io/projected/1de0b0c9-d69d-498a-9e30-37eb11daaa20-kube-api-access-sszc9\") pod \"multus-additional-cni-plugins-8fqqm\" (UID: \"1de0b0c9-d69d-498a-9e30-37eb11daaa20\") " pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.135684 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.135667    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7v2mn\" (UniqueName: \"kubernetes.io/projected/85f5d22b-9e6d-4b28-b142-871a195b0cbf-kube-api-access-7v2mn\") pod \"iptables-alerter-2kmvk\" (UID: \"85f5d22b-9e6d-4b28-b142-871a195b0cbf\") " pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.135762 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.135711    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rkll5\" (UniqueName: \"kubernetes.io/projected/5ad42386-cffd-4d60-b3e8-f1ba059b1608-kube-api-access-rkll5\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.217884 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217846    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.217884 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217890    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-systemd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217914    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-config\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217937    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-netd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-log-socket\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217969    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.217980    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75d94286-3829-46c1-8f14-3766f5477caf-ovn-node-metrics-cert\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218043    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-ovn\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218062    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-kubelet\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218086 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218076    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-bin\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218091    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-script-lib\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218113    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-slash\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-node-log\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218139    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pcbss\" (UniqueName: \"kubernetes.io/projected/75d94286-3829-46c1-8f14-3766f5477caf-kube-api-access-pcbss\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218154    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-systemd-units\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218166    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-netns\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218189    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218202    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-var-lib-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218215    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-env-overrides\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218231    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-etc-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218251    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218278    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-node-log\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218308    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218359    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-ovn-kubernetes\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218366    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-bin\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218370    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-systemd-units\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218414    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-ovn\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218419    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-run-systemd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218447    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-run-netns\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218480    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-cni-netd\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218488    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-log-socket\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218518    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-kubelet\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218526    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-host-slash\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-var-lib-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218623    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/75d94286-3829-46c1-8f14-3766f5477caf-etc-openvswitch\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.218774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-env-overrides\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.219246 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.218851    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-script-lib\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.219445 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.219412    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/75d94286-3829-46c1-8f14-3766f5477caf-ovnkube-config\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.221087 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.221067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/75d94286-3829-46c1-8f14-3766f5477caf-ovn-node-metrics-cert\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.227002 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.226955    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pcbss\" (UniqueName: \"kubernetes.io/projected/75d94286-3829-46c1-8f14-3766f5477caf-kube-api-access-pcbss\") pod \"ovnkube-node-snvlb\" (UID: \"75d94286-3829-46c1-8f14-3766f5477caf\") " pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.302863 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.302830    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-2kmvk"
Sep 10 15:42:04.310505 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.310474    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-ctnsr"
Sep 10 15:42:04.322224 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.322053    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-z5cwm"
Sep 10 15:42:04.324871 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.324852    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-p2x9j"
Sep 10 15:42:04.332512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.332488    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn"
Sep 10 15:42:04.337826 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.337800    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:04.343103 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.343084    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-8fqqm"
Sep 10 15:42:04.348800 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.348759    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:04.460830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.460806    2560 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 10 15:42:04.620414 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.620327    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:04.620414 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.620384    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:04.620586 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.620462    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.620586 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.620501    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.620586 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.620522    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.620508809 +0000 UTC m=+4.000455854 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:04.620586 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.620545    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.620531573 +0000 UTC m=+4.000478619 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:04.721170 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:04.721142    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:04.721330 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.721299    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:04.721330 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.721322    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:04.721451 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.721356    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.721451 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:04.721405    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:05.721392552 +0000 UTC m=+4.101339594 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:04.909301 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.907425    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16946fc3_49ae_4ce0_a1ca_a8e4b7c9b752.slice/crio-bfebaa956ec198751d278b6c6010019c44d44f454046623dd92dfbf4f2c7e93f WatchSource:0}: Error finding container bfebaa956ec198751d278b6c6010019c44d44f454046623dd92dfbf4f2c7e93f: Status 404 returned error can't find the container with id bfebaa956ec198751d278b6c6010019c44d44f454046623dd92dfbf4f2c7e93f
Sep 10 15:42:04.909301 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.908655    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75d94286_3829_46c1_8f14_3766f5477caf.slice/crio-72ec3a4480d3ec933e64eb7453f5db4637c99552b9b6153a8deb567a93c589d7 WatchSource:0}: Error finding container 72ec3a4480d3ec933e64eb7453f5db4637c99552b9b6153a8deb567a93c589d7: Status 404 returned error can't find the container with id 72ec3a4480d3ec933e64eb7453f5db4637c99552b9b6153a8deb567a93c589d7
Sep 10 15:42:04.929457 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.929425    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1de0b0c9_d69d_498a_9e30_37eb11daaa20.slice/crio-a505d09b7270b53e7baeb1d0636b82dd530fdfb810dac9d456c9b69bbf441a20 WatchSource:0}: Error finding container a505d09b7270b53e7baeb1d0636b82dd530fdfb810dac9d456c9b69bbf441a20: Status 404 returned error can't find the container with id a505d09b7270b53e7baeb1d0636b82dd530fdfb810dac9d456c9b69bbf441a20
Sep 10 15:42:04.930263 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.930223    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4cbf0868_ca1b_4c92_9232_9e08f6577e36.slice/crio-889a2b36553d20b817fb1bfaed4c1250cacd642c0b5bd0a667429c997a5e500e WatchSource:0}: Error finding container 889a2b36553d20b817fb1bfaed4c1250cacd642c0b5bd0a667429c997a5e500e: Status 404 returned error can't find the container with id 889a2b36553d20b817fb1bfaed4c1250cacd642c0b5bd0a667429c997a5e500e
Sep 10 15:42:04.931027 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.931011    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6c95eec8_944e_4836_b0c4_8601846bb4e5.slice/crio-c1b5a6fb15159654b0e9d2076ffc13d74f66ef1b1c4d8cb52d98ad078234afb1 WatchSource:0}: Error finding container c1b5a6fb15159654b0e9d2076ffc13d74f66ef1b1c4d8cb52d98ad078234afb1: Status 404 returned error can't find the container with id c1b5a6fb15159654b0e9d2076ffc13d74f66ef1b1c4d8cb52d98ad078234afb1
Sep 10 15:42:04.936191 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.936171    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod85f5d22b_9e6d_4b28_b142_871a195b0cbf.slice/crio-331359da46d1a261e6244f7bf21a2737d770cd6fe213db353959b2e61d1113db WatchSource:0}: Error finding container 331359da46d1a261e6244f7bf21a2737d770cd6fe213db353959b2e61d1113db: Status 404 returned error can't find the container with id 331359da46d1a261e6244f7bf21a2737d770cd6fe213db353959b2e61d1113db
Sep 10 15:42:04.937679 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:04.937658    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod79fa45c1_bfe8_4cfb_b02b_18381665958b.slice/crio-6408b5fa7d5041e0a9f40ec5218e590525e0125d14758effe9790dddd4e1b778 WatchSource:0}: Error finding container 6408b5fa7d5041e0a9f40ec5218e590525e0125d14758effe9790dddd4e1b778: Status 404 returned error can't find the container with id 6408b5fa7d5041e0a9f40ec5218e590525e0125d14758effe9790dddd4e1b778
Sep 10 15:42:05.051317 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.051278    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:03 +0000 UTC" deadline="2028-04-12 05:52:11.073474246 +0000 UTC"
Sep 10 15:42:05.051317 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.051313    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13910h10m6.022164323s"
Sep 10 15:42:05.110794 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.110762    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:05.110938 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.110870    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:05.121481 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.121451    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerStarted","Data":"a505d09b7270b53e7baeb1d0636b82dd530fdfb810dac9d456c9b69bbf441a20"}
Sep 10 15:42:05.122624 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.122597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"72ec3a4480d3ec933e64eb7453f5db4637c99552b9b6153a8deb567a93c589d7"}
Sep 10 15:42:05.123592 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.123567    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2x9j" event={"ID":"5e7b1b86-b666-4320-acf4-b0193d8eb053","Type":"ContainerStarted","Data":"44ea713cd5479c2bd5ba5b730c815dc57db219ec529d1b3e6bcd1277f77509de"}
Sep 10 15:42:05.124489 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.124470    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5cwm" event={"ID":"6c95eec8-944e-4836-b0c4-8601846bb4e5","Type":"ContainerStarted","Data":"c1b5a6fb15159654b0e9d2076ffc13d74f66ef1b1c4d8cb52d98ad078234afb1"}
Sep 10 15:42:05.125388 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.125371    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn" event={"ID":"4cbf0868-ca1b-4c92-9232-9e08f6577e36","Type":"ContainerStarted","Data":"889a2b36553d20b817fb1bfaed4c1250cacd642c0b5bd0a667429c997a5e500e"}
Sep 10 15:42:05.126362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.126325    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zjv5t" event={"ID":"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752","Type":"ContainerStarted","Data":"bfebaa956ec198751d278b6c6010019c44d44f454046623dd92dfbf4f2c7e93f"}
Sep 10 15:42:05.128179 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.128161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal" event={"ID":"17ace8c758eb0a5cfdc6b4f1e8d25f1a","Type":"ContainerStarted","Data":"8852a793154f6fd4b21ddaa4969865cc795f8d820a2f9b98a9ae3f24db4754f5"}
Sep 10 15:42:05.129373 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.129356    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctnsr" event={"ID":"79fa45c1-bfe8-4cfb-b02b-18381665958b","Type":"ContainerStarted","Data":"6408b5fa7d5041e0a9f40ec5218e590525e0125d14758effe9790dddd4e1b778"}
Sep 10 15:42:05.130277 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.130261    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kmvk" event={"ID":"85f5d22b-9e6d-4b28-b142-871a195b0cbf","Type":"ContainerStarted","Data":"331359da46d1a261e6244f7bf21a2737d770cd6fe213db353959b2e61d1113db"}
Sep 10 15:42:05.139780 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.139736    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-138-241.ec2.internal" podStartSLOduration=2.139723547 podStartE2EDuration="2.139723547s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:05.139533849 +0000 UTC m=+3.519480912" watchObservedRunningTime="2026-09-10 15:42:05.139723547 +0000 UTC m=+3.519670608"
Sep 10 15:42:05.626108 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.626046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:05.626108 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.626104    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:05.626374 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.626269    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:05.626374 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.626358    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.626316169 +0000 UTC m=+6.006263218 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:05.626784 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.626762    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:05.626881 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.626820    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.626806124 +0000 UTC m=+6.006753173 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:05.729382 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:05.727442    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:05.729382 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.727609    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:05.729382 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.727631    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:05.729382 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.727643    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:05.729382 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:05.727699    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:07.727682017 +0000 UTC m=+6.107629074 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:06.052143 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.052077    2560 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-09 15:37:03 +0000 UTC" deadline="2028-02-12 23:23:55.916707621 +0000 UTC"
Sep 10 15:42:06.052143 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.052141    2560 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12487h41m49.864571284s"
Sep 10 15:42:06.112825 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.112794    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:06.113016 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:06.112981    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:06.113016 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.113004    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:06.113134 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:06.113107    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:06.138886 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.138848    2560 generic.go:356] "Generic (PLEG): container finished" podID="a52e03e9a1cd697f3e8b90f6f1d339ef" containerID="57f12c40484eb61a4dce14b6b22a1ce9d77d09b11afd96a22488539896172365" exitCode=0
Sep 10 15:42:06.139042 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:06.138959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal" event={"ID":"a52e03e9a1cd697f3e8b90f6f1d339ef","Type":"ContainerDied","Data":"57f12c40484eb61a4dce14b6b22a1ce9d77d09b11afd96a22488539896172365"}
Sep 10 15:42:07.110723 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.110690    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:07.111216 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.110819    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:07.149376 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.149177    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal" event={"ID":"a52e03e9a1cd697f3e8b90f6f1d339ef","Type":"ContainerStarted","Data":"e61e2cbe0a41c70daafc11edd5a3391eef28b72e2924cef9dbb5ddb73e10e489"}
Sep 10 15:42:07.164436 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.163577    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-138-241.ec2.internal" podStartSLOduration=4.16355603 podStartE2EDuration="4.16355603s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:07.163157879 +0000 UTC m=+5.543104947" watchObservedRunningTime="2026-09-10 15:42:07.16355603 +0000 UTC m=+5.543503098"
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.642161    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.642207    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.642372    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.642436    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.642416545 +0000 UTC m=+10.022363591 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.642842    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:07.642926 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.642891    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.642876688 +0000 UTC m=+10.022823738 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:07.743089 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:07.743053    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:07.743274 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.743244    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:07.743274 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.743269    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:07.743402 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.743282    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:07.743402 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:07.743371    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:11.743332667 +0000 UTC m=+10.123279733 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:08.111048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:08.110561    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:08.111048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:08.110577    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:08.111048 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:08.110715    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:08.111048 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:08.110817    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:09.110971 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:09.110934    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:09.111167 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:09.111076    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:10.113214 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:10.113171    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:10.113690 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:10.113378    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:10.113787 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:10.113753    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:10.113869 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:10.113848    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:11.110961 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:11.110859    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:11.111125 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.110990    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:11.679383    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:11.679436    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.679592    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.679663    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.6796415 +0000 UTC m=+18.059588546 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.680093    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:11.680198 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.680152    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.680135307 +0000 UTC m=+18.060082351 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:11.780956 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:11.780286    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:11.780956 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.780534    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:11.780956 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.780555    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:11.780956 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.780568    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:11.780956 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:11.780632    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:19.780613606 +0000 UTC m=+18.160560673 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:12.111657 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:12.111581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:12.111825 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:12.111718    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:12.112089 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:12.112070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:12.112169 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:12.112152    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:13.110950 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:13.110433    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:13.110950 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:13.110569    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:14.110927 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:14.110892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:14.111120 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:14.111030    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:14.111120 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:14.111112    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:14.111584 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:14.111187    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:15.110485 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:15.110444    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:15.110673 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:15.110575    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:16.111073 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:16.110864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:16.111513 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:16.110854    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:16.111513 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:16.111157    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:16.111513 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:16.111250    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:17.110865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:17.110820    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:17.111025 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:17.110952    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:18.110614 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:18.110570    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:18.111062 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:18.110685    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:18.111167 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:18.111145    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:18.111283 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:18.111260    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:19.110869 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:19.110828    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:19.111362 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.110961    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:19.750695 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:19.750657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:19.750695 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:19.750703    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:19.751237 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.751212    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:19.751332 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.751301    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.75127817 +0000 UTC m=+34.131225218 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:19.751824 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.751807    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:19.751900 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.751868    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.751852816 +0000 UTC m=+34.131799867 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:19.851970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:19.851931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:19.852151 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.852132    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:19.852226 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.852161    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:19.852226 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.852173    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:19.852304 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:19.852229    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:35.852211375 +0000 UTC m=+34.232158423 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:20.110644 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:20.110552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:20.110793 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:20.110552    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:20.110793 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:20.110740    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:20.110912 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:20.110886    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:21.110108 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:21.110070    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:21.110268 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:21.110198    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:22.110849 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:22.110758    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:22.111290 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:22.110866    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:22.111290 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:22.110954    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:22.111290 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:22.111073    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:23.110547 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.110317    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:23.110661 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:23.110635    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:23.198780 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.198748    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-ctnsr" event={"ID":"79fa45c1-bfe8-4cfb-b02b-18381665958b","Type":"ContainerStarted","Data":"e5985645203d1b7eb645b790b076832af38c028fd9ab0bc4dfe9211323c9eebd"}
Sep 10 15:42:23.200270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.200244    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="25fc1d0422c39d21dd880bbac9926b74dccb04c6ce8dec10685c80deaf551bdb" exitCode=0
Sep 10 15:42:23.200455 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.200319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"25fc1d0422c39d21dd880bbac9926b74dccb04c6ce8dec10685c80deaf551bdb"}
Sep 10 15:42:23.203179 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.203155    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"9b87348482e48c58fbd363e018b9e1f74178de19031e57261fd436e460695c16"}
Sep 10 15:42:23.203278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.203235    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"5d7663858422a9d46ff8f8b0811e94999749da382858b1bc5c4bd2faf8c55e6f"}
Sep 10 15:42:23.203278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.203274    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"4de3d17c2e234a21e141ebd1d7507668e69fb086c4f15504eb6e7adefe278919"}
Sep 10 15:42:23.203427 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.203311    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"19faa49426f81bf3beedd9cafaaef397508360f481b94cea5b54ed1aa7dc85c0"}
Sep 10 15:42:23.203427 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.203360    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"c54618aaf4fc6af91ec374604f0dfd88be5ebb54a5b24cf73f0edac3ba00ecb3"}
Sep 10 15:42:23.204683 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.204583    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-p2x9j" event={"ID":"5e7b1b86-b666-4320-acf4-b0193d8eb053","Type":"ContainerStarted","Data":"a51d4866659282b78aade8d7a63229db19a907a5df7524d448fc6bcc20ed50b5"}
Sep 10 15:42:23.206354 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.206313    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-z5cwm" event={"ID":"6c95eec8-944e-4836-b0c4-8601846bb4e5","Type":"ContainerStarted","Data":"df6d25888deee6e36278e790b37e44ae1b17f43bbe6d57df201eaac07a3cfdc1"}
Sep 10 15:42:23.207874 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.207851    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn" event={"ID":"4cbf0868-ca1b-4c92-9232-9e08f6577e36","Type":"ContainerStarted","Data":"592efbaf96ed35560bce8654df84680bcc6e6fc653d139bb03a3fd63efae982d"}
Sep 10 15:42:23.209310 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.209288    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-zjv5t" event={"ID":"16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752","Type":"ContainerStarted","Data":"532a8f863592a195bc01489c6217621c78599cf1e953f52489c3b1fd504e175b"}
Sep 10 15:42:23.210278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.210226    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-ctnsr" podStartSLOduration=3.504814793 podStartE2EDuration="21.210210672s" totalImagesPullingTime="17.705395879s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.209962566 +0000 UTC m=+21.589909632" watchObservedRunningTime="2026-09-10 15:42:23.210210672 +0000 UTC m=+21.590157738"
Sep 10 15:42:23.223797 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.223740    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-zjv5t" podStartSLOduration=3.523743702 podStartE2EDuration="21.223719926s" totalImagesPullingTime="17.699976224s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.222857273 +0000 UTC m=+21.602804340" watchObservedRunningTime="2026-09-10 15:42:23.223719926 +0000 UTC m=+21.603666993"
Sep 10 15:42:23.254198 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.254157    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-p2x9j" podStartSLOduration=3.533400555 podStartE2EDuration="21.254141188s" totalImagesPullingTime="17.720740633s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.25350299 +0000 UTC m=+21.633450056" watchObservedRunningTime="2026-09-10 15:42:23.254141188 +0000 UTC m=+21.634088255"
Sep 10 15:42:23.254317 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:23.254295    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-z5cwm" podStartSLOduration=3.483424716 podStartE2EDuration="21.254285763s" totalImagesPullingTime="17.770861047s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:23.237733437 +0000 UTC m=+21.617680503" watchObservedRunningTime="2026-09-10 15:42:23.254285763 +0000 UTC m=+21.634232832"
Sep 10 15:42:24.111022 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.110995    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:24.111125 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.111036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:24.111158 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:24.111123    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:24.111218 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:24.111202    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:24.122631 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.122606    2560 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 10 15:42:24.212361 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.212257    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-2kmvk" event={"ID":"85f5d22b-9e6d-4b28-b142-871a195b0cbf","Type":"ContainerStarted","Data":"60c390a9116a7795a6121770bdf1fa28b48674c22f6c0169947259e307f0f8e3"}
Sep 10 15:42:24.215498 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.215473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"a8a01db49eb1126afcc12f51038be356aa0dd6fe4d766fe3ee3c535505e25ac5"}
Sep 10 15:42:24.217076 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.217051    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn" event={"ID":"4cbf0868-ca1b-4c92-9232-9e08f6577e36","Type":"ContainerStarted","Data":"6462c24a5100552a61b94a1e54505c7458fdcc939f147fa0dec1e614d6338a57"}
Sep 10 15:42:24.224254 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:24.224219    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-2kmvk" podStartSLOduration=5.458341319 podStartE2EDuration="23.224206765s" totalImagesPullingTime="17.765865446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:24.223849876 +0000 UTC m=+22.603796953" watchObservedRunningTime="2026-09-10 15:42:24.224206765 +0000 UTC m=+22.604153831"
Sep 10 15:42:25.076505 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.076402    2560 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-10T15:42:24.122625258Z","UUID":"bd0ba3b9-f4cb-4b51-8d9b-eaabf0f13503","Handler":null,"Name":"","Endpoint":""}
Sep 10 15:42:25.077926 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.077907    2560 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 10 15:42:25.078028 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.077935    2560 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 10 15:42:25.110487 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.110459    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:25.110656 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:25.110558    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:25.991247 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.991198    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:25.991829 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:25.991807    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:26.114542 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.114514    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:26.114740 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.114519    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:26.114740 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:26.114647    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:26.114740 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:26.114719    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:26.225390 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.225334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"c0abf31ede28a1577ba6d469baa468b7238708e899f756371d1eefaa57ded155"}
Sep 10 15:42:26.227508 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.227472    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn" event={"ID":"4cbf0868-ca1b-4c92-9232-9e08f6577e36","Type":"ContainerStarted","Data":"21ed57fdb3ad8237ef52e2c375a1c445b9db77a8e298137ff7a1cb5eb7cf0176"}
Sep 10 15:42:26.227780 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.227758    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:26.228189 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.228167    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-zjv5t"
Sep 10 15:42:26.246117 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:26.246021    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-zp5pn" podStartSLOduration=4.3178555020000005 podStartE2EDuration="24.246006643s" totalImagesPullingTime="19.928151141s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:26.245383308 +0000 UTC m=+24.625330374" watchObservedRunningTime="2026-09-10 15:42:26.246006643 +0000 UTC m=+24.625953708"
Sep 10 15:42:27.110381 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:27.110175    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:27.110872 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:27.110481    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:28.110709 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:28.110673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:28.111171 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:28.110673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:28.111171 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:28.110813    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:28.111171 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:28.110858    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:29.109876 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.109853    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:29.109983 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:29.109951    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:29.238516 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.238251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" event={"ID":"75d94286-3829-46c1-8f14-3766f5477caf","Type":"ContainerStarted","Data":"67f1eadd42b624aa6b85463c90dde91cf2aca7037a03e793e4f78e3c9b3e5d65"}
Sep 10 15:42:29.238918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.238569    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:29.238918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.238625    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:29.254174 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.254151    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:29.264136 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:29.264087    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb" podStartSLOduration=9.526901923 podStartE2EDuration="27.264070782s" totalImagesPullingTime="17.737168859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:29.26356814 +0000 UTC m=+27.643515206" watchObservedRunningTime="2026-09-10 15:42:29.264070782 +0000 UTC m=+27.644017890"
Sep 10 15:42:30.110631 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.110597    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:30.110794 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.110644    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:30.110794 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:30.110730    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:30.110903 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:30.110836    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:30.242613 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.242571    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="356dfc41dc38893ab8c789c2247b2078d9a6f55bca921c1487d6b943986e3f22" exitCode=0
Sep 10 15:42:30.243023 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.242664    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"356dfc41dc38893ab8c789c2247b2078d9a6f55bca921c1487d6b943986e3f22"}
Sep 10 15:42:30.243857 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.243189    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:30.259051 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.259022    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:42:30.897052 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.896971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drwxg"]
Sep 10 15:42:30.897203 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.897095    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:30.897258 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:30.897198    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:30.899775 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.899569    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bs58t"]
Sep 10 15:42:30.899775 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.899670    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:30.899775 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:30.899770    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:30.900841 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.900818    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-msr9m"]
Sep 10 15:42:30.900951 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:30.900915    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:30.901004 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:30.900989    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:32.248044 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:32.248007    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="5111d422ab377b2a8491be39c486b5ee353617402acda5ca5b688407c4a1a8b8" exitCode=0
Sep 10 15:42:32.248749 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:32.248077    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"5111d422ab377b2a8491be39c486b5ee353617402acda5ca5b688407c4a1a8b8"}
Sep 10 15:42:33.110142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:33.110113    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:33.110142 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:33.110136    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:33.110314 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:33.110221    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:33.110314 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:33.110224    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:33.110314 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:33.110296    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:33.110450 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:33.110382    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:34.254236 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:34.254201    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="e96424545f91548fd8e3bb53de7d2c89ff04509de9fe5c562861d4b5e4f1a3fe" exitCode=0
Sep 10 15:42:34.254699 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:34.254258    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"e96424545f91548fd8e3bb53de7d2c89ff04509de9fe5c562861d4b5e4f1a3fe"}
Sep 10 15:42:35.110972 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.110931    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:35.111178 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.110977    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:35.111178 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.110982    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:35.111178 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.111060    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-msr9m" podUID="99999ef7-513b-46a9-bc43-94c5b7ba5ab4"
Sep 10 15:42:35.111178 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.111152    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-bs58t" podUID="5ad42386-cffd-4d60-b3e8-f1ba059b1608"
Sep 10 15:42:35.111411 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.111213    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-drwxg" podUID="c7d4a7f5-53a5-49e7-85c6-4f2c375ae688"
Sep 10 15:42:35.395963 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.395881    2560 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-138-241.ec2.internal" event="NodeReady"
Sep 10 15:42:35.396410 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.396040    2560 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 10 15:42:35.432300 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.431602    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v48w7"]
Sep 10 15:42:35.449068 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.449037    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:42:35.449215 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.449176    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.453771 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.453210    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 10 15:42:35.453771 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.453251    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xpgs9\""
Sep 10 15:42:35.454370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.454128    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 10 15:42:35.465252 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.465222    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"]
Sep 10 15:42:35.465252 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.465240    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.467634 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.467611    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 10 15:42:35.467756 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.467686    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 10 15:42:35.467933 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.467911    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whst8\""
Sep 10 15:42:35.467933 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.467917    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 10 15:42:35.474397 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.473926    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 10 15:42:35.485574 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.485548    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"]
Sep 10 15:42:35.485730 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.485674    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.488177 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.488149    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 10 15:42:35.488297 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.488205    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 10 15:42:35.488437 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.488422    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 10 15:42:35.488496 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.488424    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 10 15:42:35.488551 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.488491    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-q2rbl\""
Sep 10 15:42:35.502418 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.502392    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"]
Sep 10 15:42:35.503039 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.502591    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.504963 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.504938    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 10 15:42:35.505111 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.504938    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 10 15:42:35.505256 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.505241    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 10 15:42:35.505483 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.505464    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 10 15:42:35.519962 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.519938    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v48w7"]
Sep 10 15:42:35.520114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.519970    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"]
Sep 10 15:42:35.520114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.519987    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"]
Sep 10 15:42:35.520114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.520006    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:42:35.520114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.520027    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-frshv"]
Sep 10 15:42:35.520114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.520065    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.522435 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.522404    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 10 15:42:35.532404 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.532379    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"]
Sep 10 15:42:35.532506 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.532418    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frshv"]
Sep 10 15:42:35.532506 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.532442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.535115 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.535085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 10 15:42:35.535419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.535389    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 10 15:42:35.535562 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.535426    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-8f2wr\""
Sep 10 15:42:35.535562 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.535540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 10 15:42:35.549576 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.549551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-sckvp"]
Sep 10 15:42:35.563563 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563536    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.563711 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563583    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dw626\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.563711 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563618    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b998e29-3193-4a9b-9644-944c0f55b9b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.563711 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563648    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.563832 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563789    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.563884 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563829    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2fjsj\" (UniqueName: \"kubernetes.io/projected/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-kube-api-access-2fjsj\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.563884 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563864    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.563959 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563902    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.564006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.564006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.563996    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.564082 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.564018    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.564082 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.564037    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.567293 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.567265    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.567887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.567867    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sckvp"]
Sep 10 15:42:35.570068 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.570045    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 10 15:42:35.570172 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.570085    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 10 15:42:35.570172 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.570098    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 10 15:42:35.570283 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.570098    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 10 15:42:35.570283 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.570047    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\""
Sep 10 15:42:35.664903 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.664903 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664869    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.664903 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664897    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664923    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48ec657e-a872-41bd-8a44-6eee95ca2669-klusterlet-config\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664977    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.664995    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8a7cbe-d623-4984-afaa-711821ffcbc6-cert\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665010    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ql26n\" (UniqueName: \"kubernetes.io/projected/48ec657e-a872-41bd-8a44-6eee95ca2669-kube-api-access-ql26n\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dw626\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665079    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6x5k7\" (UniqueName: \"kubernetes.io/projected/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-kube-api-access-6x5k7\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665099    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b998e29-3193-4a9b-9644-944c0f55b9b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665114    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665127    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ec657e-a872-41bd-8a44-6eee95ca2669-tmp\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665150    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665174    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-config-volume\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.665211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665216    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.665869 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.665439    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:35.665869 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.665456    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:35.665869 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.665645    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.666047 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.666025    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:36.166008972 +0000 UTC m=+34.545956016 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:35.666407 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666383    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/7b998e29-3193-4a9b-9644-944c0f55b9b3-nginx-conf\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.666407 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666377    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2fjsj\" (UniqueName: \"kubernetes.io/projected/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-kube-api-access-2fjsj\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666455    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dee925d-55b1-487a-96c1-3166a7cf3bf5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666487    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666527    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666551    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:35.666605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666576    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7sv2w\" (UniqueName: \"kubernetes.io/projected/1dee925d-55b1-487a-96c1-3166a7cf3bf5-kube-api-access-7sv2w\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666610    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8qnn\" (UniqueName: \"kubernetes.io/projected/4e8a7cbe-d623-4984-afaa-711821ffcbc6-kube-api-access-l8qnn\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666633    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-tmp-dir\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.666652    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.666724    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:36.166687178 +0000 UTC m=+34.546634238 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666758    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666779    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.666865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.666819    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.670144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.670122    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.670144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.670133    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.670289 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.670274    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.675036 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.675006    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dw626\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.676031 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.675986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:35.676424 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.676404    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2fjsj\" (UniqueName: \"kubernetes.io/projected/f616fd7e-5da9-4517-8ae7-591d93fbb1e5-kube-api-access-2fjsj\") pod \"managed-serviceaccount-addon-agent-5b784d75db-8ss74\" (UID: \"f616fd7e-5da9-4517-8ae7-591d93fbb1e5\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.767375 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767309    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8qnn\" (UniqueName: \"kubernetes.io/projected/4e8a7cbe-d623-4984-afaa-711821ffcbc6-kube-api-access-l8qnn\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.767375 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767366    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-tmp-dir\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767417    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767446    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767467    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767487    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767531    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48ec657e-a872-41bd-8a44-6eee95ca2669-klusterlet-config\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767546    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767564    2560 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:35.767618 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8a7cbe-d623-4984-afaa-711821ffcbc6-cert\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767626    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.767606405 +0000 UTC m=+66.147553463 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767636    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ql26n\" (UniqueName: \"kubernetes.io/projected/48ec657e-a872-41bd-8a44-6eee95ca2669-kube-api-access-ql26n\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767678    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:36.267660311 +0000 UTC m=+34.647607383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.767699    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret podName:c7d4a7f5-53a5-49e7-85c6-4f2c375ae688 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.767691075 +0000 UTC m=+66.147638130 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret") pod "global-pull-secret-syncer-drwxg" (UID: "c7d4a7f5-53a5-49e7-85c6-4f2c375ae688") : object "kube-system"/"original-pull-secret" not registered
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767741    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6x5k7\" (UniqueName: \"kubernetes.io/projected/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-kube-api-access-6x5k7\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767802    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767824    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ec657e-a872-41bd-8a44-6eee95ca2669-tmp\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767854    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-config-volume\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767904    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-tmp-dir\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767903    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dee925d-55b1-487a-96c1-3166a7cf3bf5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767951    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.767999    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7sv2w\" (UniqueName: \"kubernetes.io/projected/1dee925d-55b1-487a-96c1-3166a7cf3bf5-kube-api-access-7sv2w\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768895 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.768246    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/48ec657e-a872-41bd-8a44-6eee95ca2669-tmp\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.768953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.768915    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/1dee925d-55b1-487a-96c1-3166a7cf3bf5-additional-service-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.768953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.768934    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-config-volume\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.770797 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.770771    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-ca\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.771091 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.771069    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.771651 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.771629    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.771754 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.771655    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4e8a7cbe-d623-4984-afaa-711821ffcbc6-cert\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.771754 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.771676    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1dee925d-55b1-487a-96c1-3166a7cf3bf5-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.772245 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.772205    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/48ec657e-a872-41bd-8a44-6eee95ca2669-klusterlet-config\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.778827 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.778801    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ql26n\" (UniqueName: \"kubernetes.io/projected/48ec657e-a872-41bd-8a44-6eee95ca2669-kube-api-access-ql26n\") pod \"klusterlet-addon-workmgr-848687f5d7-ghsz5\" (UID: \"48ec657e-a872-41bd-8a44-6eee95ca2669\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.781236 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.781215    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7sv2w\" (UniqueName: \"kubernetes.io/projected/1dee925d-55b1-487a-96c1-3166a7cf3bf5-kube-api-access-7sv2w\") pod \"cluster-proxy-proxy-agent-6b5979b477-7hfpd\" (UID: \"1dee925d-55b1-487a-96c1-3166a7cf3bf5\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.781413 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.781255    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8qnn\" (UniqueName: \"kubernetes.io/projected/4e8a7cbe-d623-4984-afaa-711821ffcbc6-kube-api-access-l8qnn\") pod \"ingress-canary-frshv\" (UID: \"4e8a7cbe-d623-4984-afaa-711821ffcbc6\") " pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.782379 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.782355    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6x5k7\" (UniqueName: \"kubernetes.io/projected/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-kube-api-access-6x5k7\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:35.808508 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.808473    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"
Sep 10 15:42:35.818680 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.818645    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:42:35.837221 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.837140    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:35.845451 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.845427    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-frshv"
Sep 10 15:42:35.858298 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.858272    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/node-resolver-8j8vp"]
Sep 10 15:42:35.870987 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.868585    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:35.870987 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.868768    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 10 15:42:35.870987 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.868794    2560 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 10 15:42:35.870987 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.868806    2560 projected.go:196] Error preparing data for projected volume kube-api-access-ktqh2 for pod openshift-network-diagnostics/network-check-target-msr9m: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:35.870987 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:35.868885    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2 podName:99999ef7-513b-46a9-bc43-94c5b7ba5ab4 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.868864901 +0000 UTC m=+66.248811951 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-ktqh2" (UniqueName: "kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2") pod "network-check-target-msr9m" (UID: "99999ef7-513b-46a9-bc43-94c5b7ba5ab4") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 10 15:42:35.873796 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.872331    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:35.875050 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.875026    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-kqss8\""
Sep 10 15:42:35.970265 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.969956    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54ba45f6-e615-4e8a-8365-51197e8908f6-hosts-file\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:35.970265 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.970009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-72wpm\" (UniqueName: \"kubernetes.io/projected/54ba45f6-e615-4e8a-8365-51197e8908f6-kube-api-access-72wpm\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:35.970265 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:35.970129    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ba45f6-e615-4e8a-8365-51197e8908f6-tmp-dir\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.070643 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.070608    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ba45f6-e615-4e8a-8365-51197e8908f6-tmp-dir\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.070832 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.070766    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54ba45f6-e615-4e8a-8365-51197e8908f6-hosts-file\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.070832 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.070796    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-72wpm\" (UniqueName: \"kubernetes.io/projected/54ba45f6-e615-4e8a-8365-51197e8908f6-kube-api-access-72wpm\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.070943 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.070918    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/54ba45f6-e615-4e8a-8365-51197e8908f6-hosts-file\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.071022 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.071005    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/54ba45f6-e615-4e8a-8365-51197e8908f6-tmp-dir\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.080356 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.080307    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-72wpm\" (UniqueName: \"kubernetes.io/projected/54ba45f6-e615-4e8a-8365-51197e8908f6-kube-api-access-72wpm\") pod \"node-resolver-8j8vp\" (UID: \"54ba45f6-e615-4e8a-8365-51197e8908f6\") " pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.142466 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.142432    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74"]
Sep 10 15:42:36.145267 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.145241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"]
Sep 10 15:42:36.147626 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:36.147597    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf616fd7e_5da9_4517_8ae7_591d93fbb1e5.slice/crio-be4900f8bd76502c39df369af355deb7dd2a7098e59e485ab206b882eefd6e64 WatchSource:0}: Error finding container be4900f8bd76502c39df369af355deb7dd2a7098e59e485ab206b882eefd6e64: Status 404 returned error can't find the container with id be4900f8bd76502c39df369af355deb7dd2a7098e59e485ab206b882eefd6e64
Sep 10 15:42:36.148907 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:36.148881    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1dee925d_55b1_487a_96c1_3166a7cf3bf5.slice/crio-092325ced37a6d1b0d95d71c04e4cd852184e7efde06737a53b2786049a2d5b3 WatchSource:0}: Error finding container 092325ced37a6d1b0d95d71c04e4cd852184e7efde06737a53b2786049a2d5b3: Status 404 returned error can't find the container with id 092325ced37a6d1b0d95d71c04e4cd852184e7efde06737a53b2786049a2d5b3
Sep 10 15:42:36.165956 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.165893    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"]
Sep 10 15:42:36.170307 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:36.170280    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod48ec657e_a872_41bd_8a44_6eee95ca2669.slice/crio-7eb8e4f7510e7f4c8dc32c3545ecbc9955a5aa161b722385a938b4619b33ef7c WatchSource:0}: Error finding container 7eb8e4f7510e7f4c8dc32c3545ecbc9955a5aa161b722385a938b4619b33ef7c: Status 404 returned error can't find the container with id 7eb8e4f7510e7f4c8dc32c3545ecbc9955a5aa161b722385a938b4619b33ef7c
Sep 10 15:42:36.171473 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.171448    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:36.171579 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.171557    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:36.171821 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.171791    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:36.171821 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.171810    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:36.171995 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.171877    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.171856736 +0000 UTC m=+35.551803794 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:36.171995 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.171960    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:36.172107 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.172014    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.171998091 +0000 UTC m=+35.551945150 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:36.186866 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.186839    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-frshv"]
Sep 10 15:42:36.189706 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.189683    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-8j8vp"
Sep 10 15:42:36.190599 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:36.190567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e8a7cbe_d623_4984_afaa_711821ffcbc6.slice/crio-b5cdeb885bc487d4611ed763d08a4f2e4349be6b5886e09189ec1109089640ea WatchSource:0}: Error finding container b5cdeb885bc487d4611ed763d08a4f2e4349be6b5886e09189ec1109089640ea: Status 404 returned error can't find the container with id b5cdeb885bc487d4611ed763d08a4f2e4349be6b5886e09189ec1109089640ea
Sep 10 15:42:36.216424 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:42:36.216393    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54ba45f6_e615_4e8a_8365_51197e8908f6.slice/crio-11a759ff252692b81975a0bce9b8bbfd47c6dcb3dfe3f9985f07ed2978cea3c4 WatchSource:0}: Error finding container 11a759ff252692b81975a0bce9b8bbfd47c6dcb3dfe3f9985f07ed2978cea3c4: Status 404 returned error can't find the container with id 11a759ff252692b81975a0bce9b8bbfd47c6dcb3dfe3f9985f07ed2978cea3c4
Sep 10 15:42:36.260574 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.260491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8j8vp" event={"ID":"54ba45f6-e615-4e8a-8365-51197e8908f6","Type":"ContainerStarted","Data":"11a759ff252692b81975a0bce9b8bbfd47c6dcb3dfe3f9985f07ed2978cea3c4"}
Sep 10 15:42:36.262598 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.262533    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5" event={"ID":"48ec657e-a872-41bd-8a44-6eee95ca2669","Type":"ContainerStarted","Data":"7eb8e4f7510e7f4c8dc32c3545ecbc9955a5aa161b722385a938b4619b33ef7c"}
Sep 10 15:42:36.264001 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.263976    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frshv" event={"ID":"4e8a7cbe-d623-4984-afaa-711821ffcbc6","Type":"ContainerStarted","Data":"b5cdeb885bc487d4611ed763d08a4f2e4349be6b5886e09189ec1109089640ea"}
Sep 10 15:42:36.265277 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.265251    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerStarted","Data":"092325ced37a6d1b0d95d71c04e4cd852184e7efde06737a53b2786049a2d5b3"}
Sep 10 15:42:36.266852 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.266815    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74" event={"ID":"f616fd7e-5da9-4517-8ae7-591d93fbb1e5","Type":"ContainerStarted","Data":"be4900f8bd76502c39df369af355deb7dd2a7098e59e485ab206b882eefd6e64"}
Sep 10 15:42:36.272518 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:36.272494    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:36.272678 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.272663    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:36.272746 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:36.272724    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:37.272705542 +0000 UTC m=+35.652652586 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:37.111378 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.110601    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:42:37.111378 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.111081    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:42:37.117619 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.112281    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.118043    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.118255    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.118475    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.118690    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\""
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.118870    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\""
Sep 10 15:42:37.124327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.119097    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:42:37.181041 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.180998    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:37.181227 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.181117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:37.181297 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.181271    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:37.181297 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.181286    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:37.181426 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.181370    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:39.181327645 +0000 UTC m=+37.561274710 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:37.181929 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.181905    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:37.182046 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.181966    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:39.181950612 +0000 UTC m=+37.561897669 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:37.274396 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.274037    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-8j8vp" event={"ID":"54ba45f6-e615-4e8a-8365-51197e8908f6","Type":"ContainerStarted","Data":"6a2bf550052e69035ddb98196d1e5739f5419335ceb9d2ba6e5f5cfbe1e88d6e"}
Sep 10 15:42:37.282572 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.282539    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:37.283150 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.283119    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:37.283375 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:37.283181    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:39.283161794 +0000 UTC m=+37.663108854 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:37.293559 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:37.293332    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-8j8vp" podStartSLOduration=2.293308199 podStartE2EDuration="2.293308199s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:37.292068694 +0000 UTC m=+35.672015761" watchObservedRunningTime="2026-09-10 15:42:37.293308199 +0000 UTC m=+35.673255260"
Sep 10 15:42:39.203382 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:39.203322    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:39.203971 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.203634    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:39.203971 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.203711    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:43.203690597 +0000 UTC m=+41.583637647 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:39.204208 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:39.204181    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:39.204445 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.204418    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:39.204445 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.204439    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:39.204607 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.204488    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:43.204472575 +0000 UTC m=+41.584419623 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:39.304946 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:39.304907    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:39.305127 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.305077    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:39.305193 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:39.305150    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:43.30513017 +0000 UTC m=+41.685077217 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:43.238769 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:43.238714    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:43.238819    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.238896    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.238955    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.238970    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.238986    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:51.238963971 +0000 UTC m=+49.618911020 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:43.239252 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.239018    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:51.239002912 +0000 UTC m=+49.618949972 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:43.339305 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:43.339264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:43.339490 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.339466    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:43.339559 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:43.339545    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:42:51.339524972 +0000 UTC m=+49.719472018 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:42:47.303419 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.303382    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74" event={"ID":"f616fd7e-5da9-4517-8ae7-591d93fbb1e5","Type":"ContainerStarted","Data":"d4869f5ebfc7a84a4c6f2ef52bcc2f602a774c5f6085e5b191324d426df4708f"}
Sep 10 15:42:47.304866 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.304841    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5" event={"ID":"48ec657e-a872-41bd-8a44-6eee95ca2669","Type":"ContainerStarted","Data":"ac6723a10e2c6d7235793043f07ceea9da280865a98f759028ef90a00737b978"}
Sep 10 15:42:47.304956 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.304922    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:47.306893 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.306864    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5"
Sep 10 15:42:47.307955 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.307935    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="e2697efbcd0bcc586b31e538f8c02e6c08202dfc7ad543c9defd368c1cf722e6" exitCode=0
Sep 10 15:42:47.308046 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.308011    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"e2697efbcd0bcc586b31e538f8c02e6c08202dfc7ad543c9defd368c1cf722e6"}
Sep 10 15:42:47.309512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.309491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-frshv" event={"ID":"4e8a7cbe-d623-4984-afaa-711821ffcbc6","Type":"ContainerStarted","Data":"fc1da77082d0d009e87464bb000bcc4727af80fcacff4f1d08bafd8f6530c162"}
Sep 10 15:42:47.311513 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.311491    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerStarted","Data":"8c70f50b5aed80c411a3c4e7ab93ef790361800dc696dc3dad4635bf8fa2756d"}
Sep 10 15:42:47.311513 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.311527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerStarted","Data":"8405cc3fd4d65cc2d2f8169b6ecc984b7ebcb4b6e95a7f513901a2998d862787"}
Sep 10 15:42:47.311653 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.311545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerStarted","Data":"7398926a71601972ba72699bae1f0263ee2217919af60dee6589f349ddcfdb7b"}
Sep 10 15:42:47.319017 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.318976    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-5b784d75db-8ss74" podStartSLOduration=18.104649308 podStartE2EDuration="28.318963294s" totalImagesPullingTime="10.214313986s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:47.318313273 +0000 UTC m=+45.698260339" watchObservedRunningTime="2026-09-10 15:42:47.318963294 +0000 UTC m=+45.698910360"
Sep 10 15:42:47.336645 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.336604    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-848687f5d7-ghsz5" podStartSLOduration=18.128396136 podStartE2EDuration="28.336589369s" totalImagesPullingTime="10.208193233s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:47.335921864 +0000 UTC m=+45.715868931" watchObservedRunningTime="2026-09-10 15:42:47.336589369 +0000 UTC m=+45.716536435"
Sep 10 15:42:47.350262 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.350212    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-frshv" podStartSLOduration=2.17814193 podStartE2EDuration="12.350190856s" totalImagesPullingTime="10.172048926s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:47.349657421 +0000 UTC m=+45.729604486" watchObservedRunningTime="2026-09-10 15:42:47.350190856 +0000 UTC m=+45.730137924"
Sep 10 15:42:47.367899 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:47.367835    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" podStartSLOduration=18.154880641 podStartE2EDuration="28.367816045s" totalImagesPullingTime="10.212935404s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:19 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:47.367456195 +0000 UTC m=+45.747403262" watchObservedRunningTime="2026-09-10 15:42:47.367816045 +0000 UTC m=+45.747763110"
Sep 10 15:42:48.316810 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:48.316776    2560 generic.go:356] "Generic (PLEG): container finished" podID="1de0b0c9-d69d-498a-9e30-37eb11daaa20" containerID="5f8a310454fbfdc5317235e2a085e8ebb8fd292fb367ec159205ccc913074da3" exitCode=0
Sep 10 15:42:48.317240 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:48.316878    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerDied","Data":"5f8a310454fbfdc5317235e2a085e8ebb8fd292fb367ec159205ccc913074da3"}
Sep 10 15:42:49.325676 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:49.325640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" event={"ID":"1de0b0c9-d69d-498a-9e30-37eb11daaa20","Type":"ContainerStarted","Data":"a8487d43c972e5689f9a59bb95b83f7ec2f3290dd0fc81d13867996031f2932c"}
Sep 10 15:42:49.349374 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:49.349249    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-8fqqm" podStartSLOduration=9.502291485 podStartE2EDuration="47.349231618s" totalImagesPullingTime="37.846940133s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:42:49.348025106 +0000 UTC m=+47.727972171" watchObservedRunningTime="2026-09-10 15:42:49.349231618 +0000 UTC m=+47.729178684"
Sep 10 15:42:51.302539 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:51.302492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:51.302590    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.302671    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.302673    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.302692    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.302725    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.302711795 +0000 UTC m=+65.682658839 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:42:51.302960 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.302739    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.302732263 +0000 UTC m=+65.682679307 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:42:51.403139 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:42:51.403107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:42:51.403306 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.403237    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:42:51.403306 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:42:51.403301    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:07.403286782 +0000 UTC m=+65.783233826 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:43:02.268687 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:02.268656    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-snvlb"
Sep 10 15:43:07.328181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.328124    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.328218    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.328291    2560 secret.go:189] Couldn't get secret openshift-network-console/networking-console-plugin-cert: secret "networking-console-plugin-cert" not found
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.328336    2560 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.328373    2560 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-55bbbb4b7d-2sxbh: secret "image-registry-tls" not found
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.328393    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert podName:7b998e29-3193-4a9b-9644-944c0f55b9b3 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:39.328369272 +0000 UTC m=+97.708316337 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "networking-console-plugin-cert" (UniqueName: "kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert") pod "networking-console-plugin-79977f9844-v48w7" (UID: "7b998e29-3193-4a9b-9644-944c0f55b9b3") : secret "networking-console-plugin-cert" not found
Sep 10 15:43:07.328716 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.328415    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls podName:9c02ab93-786b-47e3-a3d8-02415f37a105 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:39.328403766 +0000 UTC m=+97.708350810 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls") pod "image-registry-55bbbb4b7d-2sxbh" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105") : secret "image-registry-tls" not found
Sep 10 15:43:07.429617 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.429578    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:07.429801 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.429724    2560 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 10 15:43:07.429851 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.429830    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls podName:7b644548-5bd5-4c2d-bf6d-5eb5230543c3 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:39.429812075 +0000 UTC m=+97.809759119 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls") pod "dns-default-sckvp" (UID: "7b644548-5bd5-4c2d-bf6d-5eb5230543c3") : secret "dns-default-metrics-tls" not found
Sep 10 15:43:07.832930 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.832891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:43:07.832930 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.832931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:43:07.835646 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.835628    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 10 15:43:07.835719 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.835702    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 10 15:43:07.843713 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.843696    2560 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 10 15:43:07.843765 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:07.843753    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs podName:5ad42386-cffd-4d60-b3e8-f1ba059b1608 nodeName:}" failed. No retries permitted until 2026-09-10 15:44:11.843736386 +0000 UTC m=+130.223683443 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs") pod "network-metrics-daemon-bs58t" (UID: "5ad42386-cffd-4d60-b3e8-f1ba059b1608") : secret "metrics-daemon-secret" not found
Sep 10 15:43:07.846725 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.846706    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/c7d4a7f5-53a5-49e7-85c6-4f2c375ae688-original-pull-secret\") pod \"global-pull-secret-syncer-drwxg\" (UID: \"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688\") " pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:43:07.933989 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.933944    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:43:07.936752 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.936734    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 10 15:43:07.946459 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.946440    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 10 15:43:07.957729 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:07.957705    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ktqh2\" (UniqueName: \"kubernetes.io/projected/99999ef7-513b-46a9-bc43-94c5b7ba5ab4-kube-api-access-ktqh2\") pod \"network-check-target-msr9m\" (UID: \"99999ef7-513b-46a9-bc43-94c5b7ba5ab4\") " pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:43:08.044301 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.044259    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-drwxg"
Sep 10 15:43:08.056405 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.056324    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-cthzr\""
Sep 10 15:43:08.064403 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.064383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:43:08.319511 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.319471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-drwxg"]
Sep 10 15:43:08.323813 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:08.323784    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc7d4a7f5_53a5_49e7_85c6_4f2c375ae688.slice/crio-cdf2cde75925598e58d55da7be0b77dc1a69c4d9f8602a9d61fcf62cfd9cd658 WatchSource:0}: Error finding container cdf2cde75925598e58d55da7be0b77dc1a69c4d9f8602a9d61fcf62cfd9cd658: Status 404 returned error can't find the container with id cdf2cde75925598e58d55da7be0b77dc1a69c4d9f8602a9d61fcf62cfd9cd658
Sep 10 15:43:08.341646 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.341618    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-msr9m"]
Sep 10 15:43:08.344591 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:08.344558    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99999ef7_513b_46a9_bc43_94c5b7ba5ab4.slice/crio-9e8d63443ad8fade6be25cd12f7632b0ca856b5a75cb3b6bf1a45a39c1b8139a WatchSource:0}: Error finding container 9e8d63443ad8fade6be25cd12f7632b0ca856b5a75cb3b6bf1a45a39c1b8139a: Status 404 returned error can't find the container with id 9e8d63443ad8fade6be25cd12f7632b0ca856b5a75cb3b6bf1a45a39c1b8139a
Sep 10 15:43:08.382734 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.382693    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-msr9m" event={"ID":"99999ef7-513b-46a9-bc43-94c5b7ba5ab4","Type":"ContainerStarted","Data":"9e8d63443ad8fade6be25cd12f7632b0ca856b5a75cb3b6bf1a45a39c1b8139a"}
Sep 10 15:43:08.383840 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:08.383810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drwxg" event={"ID":"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688","Type":"ContainerStarted","Data":"cdf2cde75925598e58d55da7be0b77dc1a69c4d9f8602a9d61fcf62cfd9cd658"}
Sep 10 15:43:13.404676 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:13.404640    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-msr9m" event={"ID":"99999ef7-513b-46a9-bc43-94c5b7ba5ab4","Type":"ContainerStarted","Data":"b15322c2279645105adb13058e5e8a761fe2c50c5eb666794ae8131ef2db5c34"}
Sep 10 15:43:13.405116 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:13.404741    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:43:13.406240 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:13.406217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-drwxg" event={"ID":"c7d4a7f5-53a5-49e7-85c6-4f2c375ae688","Type":"ContainerStarted","Data":"15b8a58eb34a3936738ee8fec880397f507d8e27277e82f1d7d16036250078c0"}
Sep 10 15:43:13.437724 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:13.437664    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-drwxg" podStartSLOduration=66.98203759 podStartE2EDuration="1m11.43764836s" totalImagesPullingTime="4.45561077s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:13.437424587 +0000 UTC m=+71.817371652" watchObservedRunningTime="2026-09-10 15:43:13.43764836 +0000 UTC m=+71.817595426"
Sep 10 15:43:13.438200 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:13.438170    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-msr9m" podStartSLOduration=68.007509768 podStartE2EDuration="1m12.438158832s" totalImagesPullingTime="4.430649064s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:13.420893745 +0000 UTC m=+71.800840812" watchObservedRunningTime="2026-09-10 15:43:13.438158832 +0000 UTC m=+71.818105927"
Sep 10 15:43:16.920137 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.920096    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-2pl8c"]
Sep 10 15:43:16.984141 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.984104    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2pl8c"]
Sep 10 15:43:16.984301 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.984225    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:16.986471 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.986444    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 10 15:43:16.986645 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.986629    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 10 15:43:16.986718 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.986667    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 10 15:43:16.986718 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.986672    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-89tcq\""
Sep 10 15:43:16.986821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:16.986783    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 10 15:43:17.104644 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.104601    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.104831 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.104672    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b5lcx\" (UniqueName: \"kubernetes.io/projected/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-api-access-b5lcx\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.104831 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.104696    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.104831 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.104715    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c151edf8-7c9c-4008-8149-f18da1a400b1-crio-socket\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.104831 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.104733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c151edf8-7c9c-4008-8149-f18da1a400b1-data-volume\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206140 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206046    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b5lcx\" (UniqueName: \"kubernetes.io/projected/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-api-access-b5lcx\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206140 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206090    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206140 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c151edf8-7c9c-4008-8149-f18da1a400b1-crio-socket\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206140 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206145    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c151edf8-7c9c-4008-8149-f18da1a400b1-data-volume\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.206181    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206206    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206255    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/c151edf8-7c9c-4008-8149-f18da1a400b1-crio-socket\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.206258    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.706240819 +0000 UTC m=+76.086187864 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.206302    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.206489 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.206394    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:17.706377582 +0000 UTC m=+76.086324641 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.206688 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.206581    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/c151edf8-7c9c-4008-8149-f18da1a400b1-data-volume\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.217521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.217495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-b5lcx\" (UniqueName: \"kubernetes.io/projected/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-api-access-b5lcx\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.710418 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.710372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.710624 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:17.710456    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:17.710624 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.710467    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.710624 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.710544    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.710527569 +0000 UTC m=+77.090474613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:17.710624 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.710544    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:17.710624 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:17.710585    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:18.710574753 +0000 UTC m=+77.090521796 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:18.717794 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:18.717758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:18.718254 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:18.717825    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:18.718254 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:18.717889    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:18.718254 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:18.717909    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:18.718254 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:18.717958    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.717944922 +0000 UTC m=+79.097891980 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:18.718254 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:18.717971    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:20.717964266 +0000 UTC m=+79.097911310 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:19.484747 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:19.484720    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8j8vp_54ba45f6-e615-4e8a-8365-51197e8908f6/dns-node-resolver/0.log"
Sep 10 15:43:20.684197 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:20.684165    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctnsr_79fa45c1-bfe8-4cfb-b02b-18381665958b/node-ca/0.log"
Sep 10 15:43:20.733654 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:20.733616    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:20.733807 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:20.733699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:20.733807 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:20.733766    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:20.733807 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:20.733794    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:20.733931 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:20.733837    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.733822553 +0000 UTC m=+83.113769597 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:20.733931 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:20.733856    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:24.733846562 +0000 UTC m=+83.113793610 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:21.484535 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:21.484494    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-frshv_4e8a7cbe-d623-4984-afaa-711821ffcbc6/serve-healthcheck-canary/0.log"
Sep 10 15:43:24.766731 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:24.766581    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:24.766731 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:24.766666    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:24.766731 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:24.766675    2560 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 10 15:43:24.767312 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:24.766737    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.766720433 +0000 UTC m=+91.146667476 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : configmap "kube-rbac-proxy" not found
Sep 10 15:43:24.767312 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:24.766808    2560 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:24.767312 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:43:24.766857    2560 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls podName:c151edf8-7c9c-4008-8149-f18da1a400b1 nodeName:}" failed. No retries permitted until 2026-09-10 15:43:32.766843523 +0000 UTC m=+91.146790586 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls") pod "insights-runtime-extractor-2pl8c" (UID: "c151edf8-7c9c-4008-8149-f18da1a400b1") : secret "insights-runtime-extractor-tls" not found
Sep 10 15:43:32.829638 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:32.829583    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:32.830080 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:32.829700    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:32.830282 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:32.830260    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/c151edf8-7c9c-4008-8149-f18da1a400b1-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:32.832105 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:32.832085    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/c151edf8-7c9c-4008-8149-f18da1a400b1-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-2pl8c\" (UID: \"c151edf8-7c9c-4008-8149-f18da1a400b1\") " pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:32.893448 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:32.893409    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-2pl8c"
Sep 10 15:43:33.163790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:33.163714    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-2pl8c"]
Sep 10 15:43:33.166660 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:33.166615    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc151edf8_7c9c_4008_8149_f18da1a400b1.slice/crio-badbd37bfdbeaba6bf2fa589544ef0acc9fc58af2d1bd51eb69b14cecad0290f WatchSource:0}: Error finding container badbd37bfdbeaba6bf2fa589544ef0acc9fc58af2d1bd51eb69b14cecad0290f: Status 404 returned error can't find the container with id badbd37bfdbeaba6bf2fa589544ef0acc9fc58af2d1bd51eb69b14cecad0290f
Sep 10 15:43:33.469698 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:33.469619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2pl8c" event={"ID":"c151edf8-7c9c-4008-8149-f18da1a400b1","Type":"ContainerStarted","Data":"865546d45f5b764b3e28b1e86fe869d675d459f6e971c5946febde1688d7de0a"}
Sep 10 15:43:33.469698 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:33.469676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2pl8c" event={"ID":"c151edf8-7c9c-4008-8149-f18da1a400b1","Type":"ContainerStarted","Data":"badbd37bfdbeaba6bf2fa589544ef0acc9fc58af2d1bd51eb69b14cecad0290f"}
Sep 10 15:43:34.474971 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:34.474937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2pl8c" event={"ID":"c151edf8-7c9c-4008-8149-f18da1a400b1","Type":"ContainerStarted","Data":"f38a87766ea553af37065bd958f341a6b39d33842aa226ed69516440a27d3ed8"}
Sep 10 15:43:35.484178 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:35.484140    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-2pl8c" event={"ID":"c151edf8-7c9c-4008-8149-f18da1a400b1","Type":"ContainerStarted","Data":"1883f0a4a039735b8d1ff234527e95fe36dea2d968ec814362dc6abaa02704ee"}
Sep 10 15:43:35.504198 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:35.504144    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-2pl8c" podStartSLOduration=17.485585333 podStartE2EDuration="19.504124902s" totalImagesPullingTime="2.018539569s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:35.503107966 +0000 UTC m=+93.883055031" watchObservedRunningTime="2026-09-10 15:43:35.504124902 +0000 UTC m=+93.884071971"
Sep 10 15:43:39.385452 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.385408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:43:39.385841 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.385503    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:43:39.387944 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.387923    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/7b998e29-3193-4a9b-9644-944c0f55b9b3-networking-console-plugin-cert\") pod \"networking-console-plugin-79977f9844-v48w7\" (UID: \"7b998e29-3193-4a9b-9644-944c0f55b9b3\") " pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:43:39.388068 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.388048    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"image-registry-55bbbb4b7d-2sxbh\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") " pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:43:39.486707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.486654    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:39.489071 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.489044    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/7b644548-5bd5-4c2d-bf6d-5eb5230543c3-metrics-tls\") pod \"dns-default-sckvp\" (UID: \"7b644548-5bd5-4c2d-bf6d-5eb5230543c3\") " pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:39.666217 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.666127    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-xpgs9\""
Sep 10 15:43:39.674148 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.674114    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7"
Sep 10 15:43:39.680798 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.680774    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-whst8\""
Sep 10 15:43:39.689108 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.688967    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:43:39.781031 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.780985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-z4knc\""
Sep 10 15:43:39.789048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.789014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:39.947035 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.946958    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-79977f9844-v48w7"]
Sep 10 15:43:39.949751 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:39.949727    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b998e29_3193_4a9b_9644_944c0f55b9b3.slice/crio-7ce6634fd325462c56e0468609add26699da639e7d8ff92a66bf55dc896b24f8 WatchSource:0}: Error finding container 7ce6634fd325462c56e0468609add26699da639e7d8ff92a66bf55dc896b24f8: Status 404 returned error can't find the container with id 7ce6634fd325462c56e0468609add26699da639e7d8ff92a66bf55dc896b24f8
Sep 10 15:43:39.984996 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:39.984963    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:43:39.987833 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:39.987807    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9c02ab93_786b_47e3_a3d8_02415f37a105.slice/crio-c8906e7674720fa74c44df53235f57f2c5df00e6bf57eb9a998783a53e4fe808 WatchSource:0}: Error finding container c8906e7674720fa74c44df53235f57f2c5df00e6bf57eb9a998783a53e4fe808: Status 404 returned error can't find the container with id c8906e7674720fa74c44df53235f57f2c5df00e6bf57eb9a998783a53e4fe808
Sep 10 15:43:40.057477 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.057445    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-sckvp"]
Sep 10 15:43:40.060155 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:40.060127    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b644548_5bd5_4c2d_bf6d_5eb5230543c3.slice/crio-d04bf0780be22b5728ce4bc65d6f42a5319bb6209ecda5271d3ab012219ed535 WatchSource:0}: Error finding container d04bf0780be22b5728ce4bc65d6f42a5319bb6209ecda5271d3ab012219ed535: Status 404 returned error can't find the container with id d04bf0780be22b5728ce4bc65d6f42a5319bb6209ecda5271d3ab012219ed535
Sep 10 15:43:40.506354 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.506301    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sckvp" event={"ID":"7b644548-5bd5-4c2d-bf6d-5eb5230543c3","Type":"ContainerStarted","Data":"d04bf0780be22b5728ce4bc65d6f42a5319bb6209ecda5271d3ab012219ed535"}
Sep 10 15:43:40.508365 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.508315    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" event={"ID":"9c02ab93-786b-47e3-a3d8-02415f37a105","Type":"ContainerStarted","Data":"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"}
Sep 10 15:43:40.508537 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.508386    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" event={"ID":"9c02ab93-786b-47e3-a3d8-02415f37a105","Type":"ContainerStarted","Data":"c8906e7674720fa74c44df53235f57f2c5df00e6bf57eb9a998783a53e4fe808"}
Sep 10 15:43:40.508537 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.508468    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:43:40.509979 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.509956    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7" event={"ID":"7b998e29-3193-4a9b-9644-944c0f55b9b3","Type":"ContainerStarted","Data":"7ce6634fd325462c56e0468609add26699da639e7d8ff92a66bf55dc896b24f8"}
Sep 10 15:43:40.529900 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:40.529841    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" podStartSLOduration=98.529820827 podStartE2EDuration="1m38.529820827s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:40.529398074 +0000 UTC m=+98.909345141" watchObservedRunningTime="2026-09-10 15:43:40.529820827 +0000 UTC m=+98.909767895"
Sep 10 15:43:41.515525 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:41.515482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7" event={"ID":"7b998e29-3193-4a9b-9644-944c0f55b9b3","Type":"ContainerStarted","Data":"3a71406dde4e76bba68cd2b9088dc92d56534c4329077e7645758c9d1c6b1137"}
Sep 10 15:43:41.533375 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:41.532822    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-79977f9844-v48w7" podStartSLOduration=98.525382306 podStartE2EDuration="1m39.53280233s" totalImagesPullingTime="1.007420024s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:41.53264269 +0000 UTC m=+99.912589757" watchObservedRunningTime="2026-09-10 15:43:41.53280233 +0000 UTC m=+99.912749397"
Sep 10 15:43:42.520841 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:42.520800    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sckvp" event={"ID":"7b644548-5bd5-4c2d-bf6d-5eb5230543c3","Type":"ContainerStarted","Data":"87d15aec3bcc33f42c934cfeac62bec27cd60201b94898c3e07cfb0d1af79a71"}
Sep 10 15:43:42.521246 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:42.520864    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-sckvp" event={"ID":"7b644548-5bd5-4c2d-bf6d-5eb5230543c3","Type":"ContainerStarted","Data":"9f2c63d4df10a0a4debfdb58b5984ee50f5098097aa0374cc70e3f178e80161a"}
Sep 10 15:43:42.543766 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:42.543713    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-sckvp" podStartSLOduration=65.861546429 podStartE2EDuration="1m7.543694026s" totalImagesPullingTime="1.682147597s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:42.543331069 +0000 UTC m=+100.923278134" watchObservedRunningTime="2026-09-10 15:43:42.543694026 +0000 UTC m=+100.923641092"
Sep 10 15:43:43.523691 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:43.523661    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:44.414279 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:44.414250    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-msr9m"
Sep 10 15:43:53.529131 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:53.529097    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-sckvp"
Sep 10 15:43:54.331356 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.331307    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-bwvsj"]
Sep 10 15:43:54.333542 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.333523    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.336008 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.335984    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 10 15:43:54.336333 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.336314    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 10 15:43:54.336450 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.336366    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 10 15:43:54.337181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.337163    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 10 15:43:54.337181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.337172    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-67ndx\""
Sep 10 15:43:54.337369 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.337205    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 10 15:43:54.337369 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.337225    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 10 15:43:54.393283 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393238    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-root\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393283 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393278    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nt75p\" (UniqueName: \"kubernetes.io/projected/8d2dc261-81af-4962-a13e-cdb30cc2572e-kube-api-access-nt75p\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393317    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393360    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-sys\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-tls\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393470    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-textfile\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393495    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393674 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393525    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-metrics-client-ca\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.393674 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.393557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-wtmp\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494285 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494250    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494285 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494289    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-sys\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494324    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-tls\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494372    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-textfile\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494391    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494409    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-metrics-client-ca\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-wtmp\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494444    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-sys\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494455    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-root\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494496    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-root\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.494564 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494509    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nt75p\" (UniqueName: \"kubernetes.io/projected/8d2dc261-81af-4962-a13e-cdb30cc2572e-kube-api-access-nt75p\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.495048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494586    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-wtmp\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.495048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.494868    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-textfile\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.495314 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.495091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-accelerators-collector-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.495314 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.495186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8d2dc261-81af-4962-a13e-cdb30cc2572e-metrics-client-ca\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.496847 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.496826    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.496938 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.496862    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8d2dc261-81af-4962-a13e-cdb30cc2572e-node-exporter-tls\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.502370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.502332    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nt75p\" (UniqueName: \"kubernetes.io/projected/8d2dc261-81af-4962-a13e-cdb30cc2572e-kube-api-access-nt75p\") pod \"node-exporter-bwvsj\" (UID: \"8d2dc261-81af-4962-a13e-cdb30cc2572e\") " pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.643126 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:54.643036    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-bwvsj"
Sep 10 15:43:54.652533 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:43:54.652499    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8d2dc261_81af_4962_a13e_cdb30cc2572e.slice/crio-821798bca4f989433ae1a4f1d57a6e97d9f15e961c869fdbc4dddc42110d8b22 WatchSource:0}: Error finding container 821798bca4f989433ae1a4f1d57a6e97d9f15e961c869fdbc4dddc42110d8b22: Status 404 returned error can't find the container with id 821798bca4f989433ae1a4f1d57a6e97d9f15e961c869fdbc4dddc42110d8b22
Sep 10 15:43:55.566577 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:55.566542    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwvsj" event={"ID":"8d2dc261-81af-4962-a13e-cdb30cc2572e","Type":"ContainerStarted","Data":"821798bca4f989433ae1a4f1d57a6e97d9f15e961c869fdbc4dddc42110d8b22"}
Sep 10 15:43:56.571946 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:56.571907    2560 generic.go:356] "Generic (PLEG): container finished" podID="8d2dc261-81af-4962-a13e-cdb30cc2572e" containerID="48027f1c4f69306b9b2e911565b1f3da1bd309621b828b7d2cdb0724054bb1c6" exitCode=0
Sep 10 15:43:56.572326 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:56.571997    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwvsj" event={"ID":"8d2dc261-81af-4962-a13e-cdb30cc2572e","Type":"ContainerDied","Data":"48027f1c4f69306b9b2e911565b1f3da1bd309621b828b7d2cdb0724054bb1c6"}
Sep 10 15:43:57.577677 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:57.577642    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwvsj" event={"ID":"8d2dc261-81af-4962-a13e-cdb30cc2572e","Type":"ContainerStarted","Data":"fd464a51798e7684192aa65f82ded44fdb2e6a01d294d0b7e487719ab824b4f3"}
Sep 10 15:43:57.578044 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:57.577694    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-bwvsj" event={"ID":"8d2dc261-81af-4962-a13e-cdb30cc2572e","Type":"ContainerStarted","Data":"48509ad24c35d0809b9d3b62666af0e0d9472293a8ef167ea8c3c692d6f40a8c"}
Sep 10 15:43:57.598750 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:57.598696    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-bwvsj" podStartSLOduration=2.592765247 podStartE2EDuration="3.598679211s" totalImagesPullingTime="936.583366ms" totalInitContainerRuntime="69.330598ms" isStatefulPod=true podCreationTimestamp="2026-09-10 15:43:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:43:57.597744622 +0000 UTC m=+115.977691704" watchObservedRunningTime="2026-09-10 15:43:57.598679211 +0000 UTC m=+115.978626254"
Sep 10 15:43:59.693664 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:59.693625    2560 patch_prober.go:28] interesting pod/image-registry-55bbbb4b7d-2sxbh container/registry namespace/openshift-image-registry: Liveness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 10 15:43:59.694028 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:43:59.693683    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 10 15:44:01.519922 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:01.519893    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:44:05.127207 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:05.127174    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:44:11.925841 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:11.925776    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:44:11.928253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:11.928227    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5ad42386-cffd-4d60-b3e8-f1ba059b1608-metrics-certs\") pod \"network-metrics-daemon-bs58t\" (UID: \"5ad42386-cffd-4d60-b3e8-f1ba059b1608\") " pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:44:11.933130 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:11.933108    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-b79h8\""
Sep 10 15:44:11.940255 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:11.940234    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-bs58t"
Sep 10 15:44:12.210446 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:12.210364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-bs58t"]
Sep 10 15:44:12.213576 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:44:12.213540    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ad42386_cffd_4d60_b3e8_f1ba059b1608.slice/crio-985e28dc2b1e130b547b343f009292bc8db2799edf59ae7c5b93d533648978d1 WatchSource:0}: Error finding container 985e28dc2b1e130b547b343f009292bc8db2799edf59ae7c5b93d533648978d1: Status 404 returned error can't find the container with id 985e28dc2b1e130b547b343f009292bc8db2799edf59ae7c5b93d533648978d1
Sep 10 15:44:12.629685 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:12.629647    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs58t" event={"ID":"5ad42386-cffd-4d60-b3e8-f1ba059b1608","Type":"ContainerStarted","Data":"985e28dc2b1e130b547b343f009292bc8db2799edf59ae7c5b93d533648978d1"}
Sep 10 15:44:13.635983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:13.635947    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs58t" event={"ID":"5ad42386-cffd-4d60-b3e8-f1ba059b1608","Type":"ContainerStarted","Data":"0fc6ef29e514cb650a40a6ce99ff3347a248d3634d266f85d71ae31417453a70"}
Sep 10 15:44:13.636501 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:13.636010    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-bs58t" event={"ID":"5ad42386-cffd-4d60-b3e8-f1ba059b1608","Type":"ContainerStarted","Data":"64005f4f75a370884b9e801c95c0a626788a6618f6f3570f734607f12ef4ba2f"}
Sep 10 15:44:13.651866 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:13.651781    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-bs58t" podStartSLOduration=130.786147807 podStartE2EDuration="2m11.651759543s" totalImagesPullingTime="865.611736ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:42:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:44:13.651246167 +0000 UTC m=+132.031193245" watchObservedRunningTime="2026-09-10 15:44:13.651759543 +0000 UTC m=+132.031706609"
Sep 10 15:44:15.820591 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:15.820549    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" podUID="1dee925d-55b1-487a-96c1-3166a7cf3bf5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:44:25.819823 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:25.819785    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" podUID="1dee925d-55b1-487a-96c1-3166a7cf3bf5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:44:30.146775 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.146726    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerName="registry" containerID="cri-o://cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e" gracePeriod=30
Sep 10 15:44:30.388466 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.388441    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:44:30.455170 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455079    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455170 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455126    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455170 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455185    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455228    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455251    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455286    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dw626\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455308    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token\") pod \"9c02ab93-786b-47e3-a3d8-02415f37a105\" (UID: \"9c02ab93-786b-47e3-a3d8-02415f37a105\") "
Sep 10 15:44:30.455681 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.455653    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:30.456134 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.456092    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:44:30.457858 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.457820    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626" (OuterVolumeSpecName: "kube-api-access-dw626") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "kube-api-access-dw626". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:30.457986 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.457966    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:30.458058 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.458038    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:30.458162 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.458140    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:44:30.458213 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.458197    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:44:30.466132 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.466096    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "9c02ab93-786b-47e3-a3d8-02415f37a105" (UID: "9c02ab93-786b-47e3-a3d8-02415f37a105"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:44:30.556857 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556810    2560 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-certificates\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.556857 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556854    2560 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/9c02ab93-786b-47e3-a3d8-02415f37a105-ca-trust-extracted\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.556857 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556863    2560 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-installation-pull-secrets\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.556857 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556874    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dw626\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-kube-api-access-dw626\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.557085 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556886    2560 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-bound-sa-token\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.557085 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556895    2560 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/9c02ab93-786b-47e3-a3d8-02415f37a105-image-registry-private-configuration\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.557085 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556903    2560 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/9c02ab93-786b-47e3-a3d8-02415f37a105-registry-tls\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.557085 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.556910    2560 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/9c02ab93-786b-47e3-a3d8-02415f37a105-trusted-ca\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:44:30.692537 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.692504    2560 generic.go:356] "Generic (PLEG): container finished" podID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerID="cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e" exitCode=0
Sep 10 15:44:30.692679 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.692551    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"
Sep 10 15:44:30.692679 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.692561    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" event={"ID":"9c02ab93-786b-47e3-a3d8-02415f37a105","Type":"ContainerDied","Data":"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"}
Sep 10 15:44:30.692679 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.692600    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-55bbbb4b7d-2sxbh" event={"ID":"9c02ab93-786b-47e3-a3d8-02415f37a105","Type":"ContainerDied","Data":"c8906e7674720fa74c44df53235f57f2c5df00e6bf57eb9a998783a53e4fe808"}
Sep 10 15:44:30.692679 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.692620    2560 scope.go:117] "RemoveContainer" containerID="cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"
Sep 10 15:44:30.704536 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.704516    2560 scope.go:117] "RemoveContainer" containerID="cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"
Sep 10 15:44:30.704869 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:44:30.704845    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e\": container with ID starting with cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e not found: ID does not exist" containerID="cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"
Sep 10 15:44:30.704923 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.704878    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e"} err="failed to get container status \"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e\": rpc error: code = NotFound desc = could not find container \"cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e\": container with ID starting with cec0af1acdba28b4ae69393574a9cc587ec6f63e0398042601a21b1766c3bd8e not found: ID does not exist"
Sep 10 15:44:30.715973 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.715908    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:44:30.719868 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:30.719832    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-55bbbb4b7d-2sxbh"]
Sep 10 15:44:32.114899 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:32.114870    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" path="/var/lib/kubelet/pods/9c02ab93-786b-47e3-a3d8-02415f37a105/volumes"
Sep 10 15:44:35.819718 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:35.819682    2560 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" podUID="1dee925d-55b1-487a-96c1-3166a7cf3bf5" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 10 15:44:35.820163 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:35.819744    2560 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd"
Sep 10 15:44:35.820203 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:35.820177    2560 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"8c70f50b5aed80c411a3c4e7ab93ef790361800dc696dc3dad4635bf8fa2756d"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 10 15:44:35.820237 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:35.820212    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" podUID="1dee925d-55b1-487a-96c1-3166a7cf3bf5" containerName="service-proxy" containerID="cri-o://8c70f50b5aed80c411a3c4e7ab93ef790361800dc696dc3dad4635bf8fa2756d" gracePeriod=30
Sep 10 15:44:36.716865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:36.716828    2560 generic.go:356] "Generic (PLEG): container finished" podID="1dee925d-55b1-487a-96c1-3166a7cf3bf5" containerID="8c70f50b5aed80c411a3c4e7ab93ef790361800dc696dc3dad4635bf8fa2756d" exitCode=2
Sep 10 15:44:36.717043 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:36.716886    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerDied","Data":"8c70f50b5aed80c411a3c4e7ab93ef790361800dc696dc3dad4635bf8fa2756d"}
Sep 10 15:44:36.717043 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:44:36.716931    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-6b5979b477-7hfpd" event={"ID":"1dee925d-55b1-487a-96c1-3166a7cf3bf5","Type":"ContainerStarted","Data":"3c14cd0dc699cd3ed52c49ffe9b78629f3b49d8e26e31cf9666c23810414a546"}
Sep 10 15:47:02.014669 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:02.014630    2560 kubelet.go:1648] "Image garbage collection succeeded"
Sep 10 15:47:19.741457 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.741423    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"]
Sep 10 15:47:19.743734 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.741688    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerName="registry"
Sep 10 15:47:19.743734 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.741700    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerName="registry"
Sep 10 15:47:19.743734 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.741742    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9c02ab93-786b-47e3-a3d8-02415f37a105" containerName="registry"
Sep 10 15:47:19.744596 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.744581    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.747018 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.746994    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 10 15:47:19.747130 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.747026    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 10 15:47:19.747130 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.746997    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-s4hf4\""
Sep 10 15:47:19.747130 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.746996    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 10 15:47:19.747276 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.747032    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 10 15:47:19.762268 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.762241    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"]
Sep 10 15:47:19.820911 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.820862    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.820911 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.820909    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qspw7\" (UniqueName: \"kubernetes.io/projected/1e9c74bf-07f2-429e-9e3e-968e6dffd734-kube-api-access-qspw7\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.821123 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.820975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.821123 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.821016    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.821123 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.821044    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922239 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qspw7\" (UniqueName: \"kubernetes.io/projected/1e9c74bf-07f2-429e-9e3e-968e6dffd734-kube-api-access-qspw7\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922433 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922258    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922433 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922297    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922433 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922334    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922597 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922500    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.922798 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.922768    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-manifests\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.924792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.924766    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/1e9c74bf-07f2-429e-9e3e-968e6dffd734-tmp\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.924941 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.924926    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-apiservice-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.925007 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.924948    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/1e9c74bf-07f2-429e-9e3e-968e6dffd734-webhook-cert\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:19.930958 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:19.930929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qspw7\" (UniqueName: \"kubernetes.io/projected/1e9c74bf-07f2-429e-9e3e-968e6dffd734-kube-api-access-qspw7\") pod \"opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf\" (UID: \"1e9c74bf-07f2-429e-9e3e-968e6dffd734\") " pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:20.055103 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:20.054999    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:20.339573 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:20.339488    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"]
Sep 10 15:47:20.343189 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:47:20.343162    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e9c74bf_07f2_429e_9e3e_968e6dffd734.slice/crio-957674044dbc0b961f809aebe34984244e516af921b3bc9ebf1e4ecaeaf8c8e0 WatchSource:0}: Error finding container 957674044dbc0b961f809aebe34984244e516af921b3bc9ebf1e4ecaeaf8c8e0: Status 404 returned error can't find the container with id 957674044dbc0b961f809aebe34984244e516af921b3bc9ebf1e4ecaeaf8c8e0
Sep 10 15:47:20.344976 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:20.344958    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:47:21.250569 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:21.250527    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf" event={"ID":"1e9c74bf-07f2-429e-9e3e-968e6dffd734","Type":"ContainerStarted","Data":"957674044dbc0b961f809aebe34984244e516af921b3bc9ebf1e4ecaeaf8c8e0"}
Sep 10 15:47:24.263843 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:24.263803    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf" event={"ID":"1e9c74bf-07f2-429e-9e3e-968e6dffd734","Type":"ContainerStarted","Data":"9520414cdbc88ee0f7e67c181ab9fa46c4ca02da7471cc5169e653cf927b3611"}
Sep 10 15:47:25.269239 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:25.269205    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e9c74bf-07f2-429e-9e3e-968e6dffd734" containerID="9520414cdbc88ee0f7e67c181ab9fa46c4ca02da7471cc5169e653cf927b3611" exitCode=0
Sep 10 15:47:25.269636 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:25.269289    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf" event={"ID":"1e9c74bf-07f2-429e-9e3e-968e6dffd734","Type":"ContainerDied","Data":"9520414cdbc88ee0f7e67c181ab9fa46c4ca02da7471cc5169e653cf927b3611"}
Sep 10 15:47:26.274169 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:26.274136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf" event={"ID":"1e9c74bf-07f2-429e-9e3e-968e6dffd734","Type":"ContainerStarted","Data":"ff6b356f60a0a460c07fd5e5bb45e029f2309c726390c80a37989a4803763f1a"}
Sep 10 15:47:26.274549 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:26.274257    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:26.297503 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:26.297455    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf" podStartSLOduration=2.800395326 podStartE2EDuration="7.297439348s" totalImagesPullingTime="3.43607004s" totalInitContainerRuntime="1.060973982s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:19 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:26.295980227 +0000 UTC m=+324.675927294" watchObservedRunningTime="2026-09-10 15:47:26.297439348 +0000 UTC m=+324.677386414"
Sep 10 15:47:37.279633 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:37.279599    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf"
Sep 10 15:47:38.353539 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.353499    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"]
Sep 10 15:47:38.362856 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.362832    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.366545 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.366522    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 10 15:47:38.367047 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.367026    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 10 15:47:38.367047 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.367042    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 10 15:47:38.367047 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.367058    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 10 15:47:38.367278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.367029    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-4fq4m\""
Sep 10 15:47:38.367278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.367034    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 10 15:47:38.369689 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.369471    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"]
Sep 10 15:47:38.445976 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.445946    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.445976 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.445980    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2nrrt\" (UniqueName: \"kubernetes.io/projected/4508ba2a-caab-4822-b231-80b4211bf721-kube-api-access-2nrrt\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.446177 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.446014    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4508ba2a-caab-4822-b231-80b4211bf721-manager-config\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.446177 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.446070    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.546601 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.546561    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.546601 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.546605    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2nrrt\" (UniqueName: \"kubernetes.io/projected/4508ba2a-caab-4822-b231-80b4211bf721-kube-api-access-2nrrt\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.546843 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.546650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4508ba2a-caab-4822-b231-80b4211bf721-manager-config\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.546843 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.546675    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.547366 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.547326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/4508ba2a-caab-4822-b231-80b4211bf721-manager-config\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.549259 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.549232    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-metrics-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.549382 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.549279    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/4508ba2a-caab-4822-b231-80b4211bf721-cert\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.562865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.562839    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2nrrt\" (UniqueName: \"kubernetes.io/projected/4508ba2a-caab-4822-b231-80b4211bf721-kube-api-access-2nrrt\") pod \"lws-controller-manager-f8dc47f5d-wqsj4\" (UID: \"4508ba2a-caab-4822-b231-80b4211bf721\") " pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.672305 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.672215    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:38.968028 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:38.967996    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"]
Sep 10 15:47:38.969666 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:47:38.969642    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4508ba2a_caab_4822_b231_80b4211bf721.slice/crio-f2ea9303d4691bdd844e23bf7ce924e01f3bdec138de6f27700c55c04068110f WatchSource:0}: Error finding container f2ea9303d4691bdd844e23bf7ce924e01f3bdec138de6f27700c55c04068110f: Status 404 returned error can't find the container with id f2ea9303d4691bdd844e23bf7ce924e01f3bdec138de6f27700c55c04068110f
Sep 10 15:47:39.111794 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.111763    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"]
Sep 10 15:47:39.114567 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.114544    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.116844 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.116819    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 10 15:47:39.116966 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.116853    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-vjsdp\""
Sep 10 15:47:39.117135 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.117118    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 10 15:47:39.117187 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.117168    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 10 15:47:39.117623 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.117586    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 10 15:47:39.130604 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.130581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"]
Sep 10 15:47:39.252060 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.252030    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-srxzw\" (UniqueName: \"kubernetes.io/projected/740e90ae-dfe3-4404-9ab5-789b28590415-kube-api-access-srxzw\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.252060 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.252063    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e90ae-dfe3-4404-9ab5-789b28590415-tmp\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.252264 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.252110    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740e90ae-dfe3-4404-9ab5-789b28590415-tls-certs\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.320612 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.320578    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4" event={"ID":"4508ba2a-caab-4822-b231-80b4211bf721","Type":"ContainerStarted","Data":"f2ea9303d4691bdd844e23bf7ce924e01f3bdec138de6f27700c55c04068110f"}
Sep 10 15:47:39.353155 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.353130    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-srxzw\" (UniqueName: \"kubernetes.io/projected/740e90ae-dfe3-4404-9ab5-789b28590415-kube-api-access-srxzw\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.353361 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.353171    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e90ae-dfe3-4404-9ab5-789b28590415-tmp\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.353361 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.353212    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740e90ae-dfe3-4404-9ab5-789b28590415-tls-certs\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.355587 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.355569    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/740e90ae-dfe3-4404-9ab5-789b28590415-tmp\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.355865 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.355701    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/740e90ae-dfe3-4404-9ab5-789b28590415-tls-certs\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.363029 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.363003    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-srxzw\" (UniqueName: \"kubernetes.io/projected/740e90ae-dfe3-4404-9ab5-789b28590415-kube-api-access-srxzw\") pod \"kube-auth-proxy-6896b79569-dmqnm\" (UID: \"740e90ae-dfe3-4404-9ab5-789b28590415\") " pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.424576 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.424543    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"
Sep 10 15:47:39.693830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:39.693728    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6896b79569-dmqnm"]
Sep 10 15:47:39.696913 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:47:39.696875    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod740e90ae_dfe3_4404_9ab5_789b28590415.slice/crio-778fa0d9412118ba527d9f7e7ac4eef093310752fc78ccc01b10a9f91bb35e8b WatchSource:0}: Error finding container 778fa0d9412118ba527d9f7e7ac4eef093310752fc78ccc01b10a9f91bb35e8b: Status 404 returned error can't find the container with id 778fa0d9412118ba527d9f7e7ac4eef093310752fc78ccc01b10a9f91bb35e8b
Sep 10 15:47:40.326297 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:40.326263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm" event={"ID":"740e90ae-dfe3-4404-9ab5-789b28590415","Type":"ContainerStarted","Data":"778fa0d9412118ba527d9f7e7ac4eef093310752fc78ccc01b10a9f91bb35e8b"}
Sep 10 15:47:43.339626 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:43.339582    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4" event={"ID":"4508ba2a-caab-4822-b231-80b4211bf721","Type":"ContainerStarted","Data":"2a55d0ed1cbf69e72c679939c7440e0c6f19d8b5e816017615d37120c1d6a321"}
Sep 10 15:47:43.340068 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:43.339668    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:47:43.341176 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:43.341151    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm" event={"ID":"740e90ae-dfe3-4404-9ab5-789b28590415","Type":"ContainerStarted","Data":"7e4e96ede0d8a973c8d5e8ce7437c43c74289aad613392ba19b2745293e28fcf"}
Sep 10 15:47:43.357532 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:43.357478    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4" podStartSLOduration=1.361638051 podStartE2EDuration="5.357464703s" totalImagesPullingTime="3.995826652s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:43.356167427 +0000 UTC m=+341.736114493" watchObservedRunningTime="2026-09-10 15:47:43.357464703 +0000 UTC m=+341.737411769"
Sep 10 15:47:43.373173 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:43.373120    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6896b79569-dmqnm" podStartSLOduration=1.056332903 podStartE2EDuration="4.373102585s" totalImagesPullingTime="3.316769682s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:47:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:47:43.372957473 +0000 UTC m=+341.752904540" watchObservedRunningTime="2026-09-10 15:47:43.373102585 +0000 UTC m=+341.753049653"
Sep 10 15:47:54.346771 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:47:54.346672    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-f8dc47f5d-wqsj4"
Sep 10 15:49:18.807529 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.807495    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"]
Sep 10 15:49:18.810478 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.810463    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:18.813011 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.812985    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Sep 10 15:49:18.813199 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.813184    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-bphzg\""
Sep 10 15:49:18.813280 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.813188    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Sep 10 15:49:18.814114 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.814099    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 10 15:49:18.814165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.814114    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 10 15:49:18.825474 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.825449    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"]
Sep 10 15:49:18.974190 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.974149    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21229c4c-dace-4cc7-b9a7-ab505624b53f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:18.974393 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.974206    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czppc\" (UniqueName: \"kubernetes.io/projected/21229c4c-dace-4cc7-b9a7-ab505624b53f-kube-api-access-czppc\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:18.974393 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:18.974265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21229c4c-dace-4cc7-b9a7-ab505624b53f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.075263 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.075180    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21229c4c-dace-4cc7-b9a7-ab505624b53f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.075263 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.075241    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czppc\" (UniqueName: \"kubernetes.io/projected/21229c4c-dace-4cc7-b9a7-ab505624b53f-kube-api-access-czppc\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.075473 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.075275    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21229c4c-dace-4cc7-b9a7-ab505624b53f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.075953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.075928    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/21229c4c-dace-4cc7-b9a7-ab505624b53f-nginx-conf\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.077761 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.077733    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/21229c4c-dace-4cc7-b9a7-ab505624b53f-plugin-serving-cert\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.088770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.088748    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-czppc\" (UniqueName: \"kubernetes.io/projected/21229c4c-dace-4cc7-b9a7-ab505624b53f-kube-api-access-czppc\") pod \"kuadrant-console-plugin-fbdd4f445-hr9gt\" (UID: \"21229c4c-dace-4cc7-b9a7-ab505624b53f\") " pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.119136 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.119109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"
Sep 10 15:49:19.404938 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.404864    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt"]
Sep 10 15:49:19.408307 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:49:19.408267    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod21229c4c_dace_4cc7_b9a7_ab505624b53f.slice/crio-82297bb5c1362e0d2e5888e1e6aa7387f3f9924566f626f39416bc9cd775a36b WatchSource:0}: Error finding container 82297bb5c1362e0d2e5888e1e6aa7387f3f9924566f626f39416bc9cd775a36b: Status 404 returned error can't find the container with id 82297bb5c1362e0d2e5888e1e6aa7387f3f9924566f626f39416bc9cd775a36b
Sep 10 15:49:19.678595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:19.678502    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt" event={"ID":"21229c4c-dace-4cc7-b9a7-ab505624b53f","Type":"ContainerStarted","Data":"82297bb5c1362e0d2e5888e1e6aa7387f3f9924566f626f39416bc9cd775a36b"}
Sep 10 15:49:24.699847 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:24.699810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt" event={"ID":"21229c4c-dace-4cc7-b9a7-ab505624b53f","Type":"ContainerStarted","Data":"0194d1017ae73ad1f7b98e14302d6dcde6948cbde98ad1f523631ee8e7f6a07d"}
Sep 10 15:49:24.730264 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:49:24.730215    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-fbdd4f445-hr9gt" podStartSLOduration=2.27548444 podStartE2EDuration="6.73020171s" totalImagesPullingTime="4.45471727s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:49:18 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:49:24.729446727 +0000 UTC m=+443.109393794" watchObservedRunningTime="2026-09-10 15:49:24.73020171 +0000 UTC m=+443.110148775"
Sep 10 15:50:06.435577 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.435540    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:50:06.455234 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.455200    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:50:06.455443 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.455323    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.458327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.458298    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 10 15:50:06.466129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.466103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:50:06.527425 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.527391    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b44eee6-bad7-4137-b352-09830839ae26-config-file\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.527425 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.527436    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbcn7\" (UniqueName: \"kubernetes.io/projected/8b44eee6-bad7-4137-b352-09830839ae26-kube-api-access-cbcn7\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.628054 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.628020    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b44eee6-bad7-4137-b352-09830839ae26-config-file\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.628188 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.628064    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbcn7\" (UniqueName: \"kubernetes.io/projected/8b44eee6-bad7-4137-b352-09830839ae26-kube-api-access-cbcn7\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.628676 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.628654    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/8b44eee6-bad7-4137-b352-09830839ae26-config-file\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.637660 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.637637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbcn7\" (UniqueName: \"kubernetes.io/projected/8b44eee6-bad7-4137-b352-09830839ae26-kube-api-access-cbcn7\") pod \"limitador-limitador-84994f7ffc-jwkr8\" (UID: \"8b44eee6-bad7-4137-b352-09830839ae26\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.761982 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.761952    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:06.765218 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.765195    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:06.765218 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.765214    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:06.767512 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.767492    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fswpm\""
Sep 10 15:50:06.772649 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.772515    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:06.829833 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.829788    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcg4m\" (UniqueName: \"kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m\") pod \"authorino-7bcfd8f9cd-9jxln\" (UID: \"60d1ff10-9015-458e-82e7-085b477284f2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:06.930447 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.930406    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcg4m\" (UniqueName: \"kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m\") pod \"authorino-7bcfd8f9cd-9jxln\" (UID: \"60d1ff10-9015-458e-82e7-085b477284f2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:06.939009 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:06.938979    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcg4m\" (UniqueName: \"kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m\") pod \"authorino-7bcfd8f9cd-9jxln\" (UID: \"60d1ff10-9015-458e-82e7-085b477284f2\") " pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:07.036949 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:07.036880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:50:07.039776 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:50:07.039750    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b44eee6_bad7_4137_b352_09830839ae26.slice/crio-4ccc7c34daaf8dce91be0f0f9a8c30dfb462e9051253bee44508ea6c45e68f99 WatchSource:0}: Error finding container 4ccc7c34daaf8dce91be0f0f9a8c30dfb462e9051253bee44508ea6c45e68f99: Status 404 returned error can't find the container with id 4ccc7c34daaf8dce91be0f0f9a8c30dfb462e9051253bee44508ea6c45e68f99
Sep 10 15:50:07.088214 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:07.088182    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:07.359239 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:07.359153    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:07.361924 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:50:07.361894    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod60d1ff10_9015_458e_82e7_085b477284f2.slice/crio-a62677460cc8e7e45a9c04d758ca80dddbd85cb01a40d46a2eff15a5ce07fcfa WatchSource:0}: Error finding container a62677460cc8e7e45a9c04d758ca80dddbd85cb01a40d46a2eff15a5ce07fcfa: Status 404 returned error can't find the container with id a62677460cc8e7e45a9c04d758ca80dddbd85cb01a40d46a2eff15a5ce07fcfa
Sep 10 15:50:07.856335 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:07.856296    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" event={"ID":"60d1ff10-9015-458e-82e7-085b477284f2","Type":"ContainerStarted","Data":"a62677460cc8e7e45a9c04d758ca80dddbd85cb01a40d46a2eff15a5ce07fcfa"}
Sep 10 15:50:07.857436 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:07.857414    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8" event={"ID":"8b44eee6-bad7-4137-b352-09830839ae26","Type":"ContainerStarted","Data":"4ccc7c34daaf8dce91be0f0f9a8c30dfb462e9051253bee44508ea6c45e68f99"}
Sep 10 15:50:09.958953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:09.958920    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:10.873961 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.873922    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" event={"ID":"60d1ff10-9015-458e-82e7-085b477284f2","Type":"ContainerStarted","Data":"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"}
Sep 10 15:50:10.874160 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.873975    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" podUID="60d1ff10-9015-458e-82e7-085b477284f2" containerName="authorino" containerID="cri-o://a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654" gracePeriod=30
Sep 10 15:50:10.879134 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.879096    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8" event={"ID":"8b44eee6-bad7-4137-b352-09830839ae26","Type":"ContainerStarted","Data":"340d70d7da997ae57223951672ef37b1ee7ef52c08be73621b18572643fa241e"}
Sep 10 15:50:10.879461 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.879432    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:10.892511 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.892463    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" podStartSLOduration=1.814876061 podStartE2EDuration="4.892447122s" totalImagesPullingTime="3.077571061s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:10.891867979 +0000 UTC m=+489.271815047" watchObservedRunningTime="2026-09-10 15:50:10.892447122 +0000 UTC m=+489.272394188"
Sep 10 15:50:10.909062 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:10.909006    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8" podStartSLOduration=1.510537467 podStartE2EDuration="4.908987077s" totalImagesPullingTime="3.39844961s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:50:06 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:50:10.907572648 +0000 UTC m=+489.287519727" watchObservedRunningTime="2026-09-10 15:50:10.908987077 +0000 UTC m=+489.288934160"
Sep 10 15:50:11.115180 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.115156    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:11.164096 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.164000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dcg4m\" (UniqueName: \"kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m\") pod \"60d1ff10-9015-458e-82e7-085b477284f2\" (UID: \"60d1ff10-9015-458e-82e7-085b477284f2\") "
Sep 10 15:50:11.166377 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.166329    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m" (OuterVolumeSpecName: "kube-api-access-dcg4m") pod "60d1ff10-9015-458e-82e7-085b477284f2" (UID: "60d1ff10-9015-458e-82e7-085b477284f2"). InnerVolumeSpecName "kube-api-access-dcg4m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:11.264892 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.264855    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcg4m\" (UniqueName: \"kubernetes.io/projected/60d1ff10-9015-458e-82e7-085b477284f2-kube-api-access-dcg4m\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:50:11.883417 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.883383    2560 generic.go:356] "Generic (PLEG): container finished" podID="60d1ff10-9015-458e-82e7-085b477284f2" containerID="a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654" exitCode=0
Sep 10 15:50:11.883615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.883427    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln"
Sep 10 15:50:11.883615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.883479    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" event={"ID":"60d1ff10-9015-458e-82e7-085b477284f2","Type":"ContainerDied","Data":"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"}
Sep 10 15:50:11.883615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.883525    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-9jxln" event={"ID":"60d1ff10-9015-458e-82e7-085b477284f2","Type":"ContainerDied","Data":"a62677460cc8e7e45a9c04d758ca80dddbd85cb01a40d46a2eff15a5ce07fcfa"}
Sep 10 15:50:11.883615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.883567    2560 scope.go:117] "RemoveContainer" containerID="a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"
Sep 10 15:50:11.892682 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.892658    2560 scope.go:117] "RemoveContainer" containerID="a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"
Sep 10 15:50:11.892982 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:50:11.892961    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654\": container with ID starting with a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654 not found: ID does not exist" containerID="a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"
Sep 10 15:50:11.893025 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.892991    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654"} err="failed to get container status \"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654\": rpc error: code = NotFound desc = could not find container \"a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654\": container with ID starting with a1e16e27dfe07f4ccfba1fced77ed36de2e0ae7772e8dd327ccba57e8cff1654 not found: ID does not exist"
Sep 10 15:50:11.903328 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.903300    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:11.907072 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:11.907046    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-9jxln"]
Sep 10 15:50:12.114376 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:12.114325    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60d1ff10-9015-458e-82e7-085b477284f2" path="/var/lib/kubelet/pods/60d1ff10-9015-458e-82e7-085b477284f2/volumes"
Sep 10 15:50:21.885132 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:21.885096    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"
Sep 10 15:50:54.680703 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.680623    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-557f876f69-mmckg"]
Sep 10 15:50:54.681124 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.680873    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60d1ff10-9015-458e-82e7-085b477284f2" containerName="authorino"
Sep 10 15:50:54.681124 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.680884    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="60d1ff10-9015-458e-82e7-085b477284f2" containerName="authorino"
Sep 10 15:50:54.681124 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.680932    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="60d1ff10-9015-458e-82e7-085b477284f2" containerName="authorino"
Sep 10 15:50:54.682793 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.682776    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:54.685085 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.685061    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fswpm\""
Sep 10 15:50:54.690187 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.690166    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-557f876f69-mmckg"]
Sep 10 15:50:54.771676 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.771634    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lp58j\" (UniqueName: \"kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j\") pod \"authorino-557f876f69-mmckg\" (UID: \"762038c8-4013-40a9-a504-c1e1879f029a\") " pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:54.776141 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.776112    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-mmckg"]
Sep 10 15:50:54.776385 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:50:54.776361    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-lp58j], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-557f876f69-mmckg" podUID="762038c8-4013-40a9-a504-c1e1879f029a"
Sep 10 15:50:54.805820 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.805785    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qsvpv"]
Sep 10 15:50:54.808306 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.808287    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:54.810703 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.810685    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:50:54.818352 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.818319    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qsvpv"]
Sep 10 15:50:54.872887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.872859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lp58j\" (UniqueName: \"kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j\") pod \"authorino-557f876f69-mmckg\" (UID: \"762038c8-4013-40a9-a504-c1e1879f029a\") " pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:54.881266 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.881234    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lp58j\" (UniqueName: \"kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j\") pod \"authorino-557f876f69-mmckg\" (UID: \"762038c8-4013-40a9-a504-c1e1879f029a\") " pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:54.916724 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.916695    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qsvpv"]
Sep 10 15:50:54.916937 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:50:54.916917    2560 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-p4xqk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6d4dfc649f-qsvpv" podUID="858a1e43-da6d-494b-92fb-885d2ccc33c5"
Sep 10 15:50:54.973411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.973380    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:54.973544 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:54.973431    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.040671 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.040634    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.040848 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.040638    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:55.046533 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.046513    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.049367 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.049330    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:55.073855 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.073828    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.074024 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.073873    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.076428 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.076400    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.082104 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.082075    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk\") pod \"authorino-6d4dfc649f-qsvpv\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") " pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:55.174525 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.174485    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lp58j\" (UniqueName: \"kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j\") pod \"762038c8-4013-40a9-a504-c1e1879f029a\" (UID: \"762038c8-4013-40a9-a504-c1e1879f029a\") "
Sep 10 15:50:55.174719 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.174540    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk\") pod \"858a1e43-da6d-494b-92fb-885d2ccc33c5\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") "
Sep 10 15:50:55.174719 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.174582    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert\") pod \"858a1e43-da6d-494b-92fb-885d2ccc33c5\" (UID: \"858a1e43-da6d-494b-92fb-885d2ccc33c5\") "
Sep 10 15:50:55.176924 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.176886    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert" (OuterVolumeSpecName: "tls-cert") pod "858a1e43-da6d-494b-92fb-885d2ccc33c5" (UID: "858a1e43-da6d-494b-92fb-885d2ccc33c5"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:50:55.176924 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.176916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk" (OuterVolumeSpecName: "kube-api-access-p4xqk") pod "858a1e43-da6d-494b-92fb-885d2ccc33c5" (UID: "858a1e43-da6d-494b-92fb-885d2ccc33c5"). InnerVolumeSpecName "kube-api-access-p4xqk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:55.177079 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.176951    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j" (OuterVolumeSpecName: "kube-api-access-lp58j") pod "762038c8-4013-40a9-a504-c1e1879f029a" (UID: "762038c8-4013-40a9-a504-c1e1879f029a"). InnerVolumeSpecName "kube-api-access-lp58j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:50:55.275628 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.275536    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lp58j\" (UniqueName: \"kubernetes.io/projected/762038c8-4013-40a9-a504-c1e1879f029a-kube-api-access-lp58j\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:50:55.275628 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.275576    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p4xqk\" (UniqueName: \"kubernetes.io/projected/858a1e43-da6d-494b-92fb-885d2ccc33c5-kube-api-access-p4xqk\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:50:55.275628 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:55.275589    2560 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/858a1e43-da6d-494b-92fb-885d2ccc33c5-tls-cert\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:50:56.044418 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.044383    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-557f876f69-mmckg"
Sep 10 15:50:56.044893 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.044384    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6d4dfc649f-qsvpv"
Sep 10 15:50:56.077743 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.077719    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qsvpv"]
Sep 10 15:50:56.081128 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.081105    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6d4dfc649f-qsvpv"]
Sep 10 15:50:56.102667 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.102639    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-557f876f69-mmckg"]
Sep 10 15:50:56.107135 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.107115    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-557f876f69-mmckg"]
Sep 10 15:50:56.114584 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.114564    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="762038c8-4013-40a9-a504-c1e1879f029a" path="/var/lib/kubelet/pods/762038c8-4013-40a9-a504-c1e1879f029a/volumes"
Sep 10 15:50:56.114801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:50:56.114788    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="858a1e43-da6d-494b-92fb-885d2ccc33c5" path="/var/lib/kubelet/pods/858a1e43-da6d-494b-92fb-885d2ccc33c5/volumes"
Sep 10 15:51:38.740011 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:51:38.739962    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:01.591745 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:01.591711    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:09.357768 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.357722    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"]
Sep 10 15:52:09.360032 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.360014    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.363171 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.363155    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-h87hn\""
Sep 10 15:52:09.363329 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.363313    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 10 15:52:09.364259 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.364248    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 10 15:52:09.364308 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.364256    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 10 15:52:09.373667 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.373570    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"]
Sep 10 15:52:09.478109 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478066    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.478109 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478112    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hh7m4\" (UniqueName: \"kubernetes.io/projected/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-kube-api-access-hh7m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.478370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478134    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.478370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478221    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.478370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478279    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.478370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.478321    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579255 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579219    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579264    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hh7m4\" (UniqueName: \"kubernetes.io/projected/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-kube-api-access-hh7m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579308    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579331    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579464    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579783 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579734    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579783 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579754    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.579886 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.579797    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.581763 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.581741    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.582036 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.582021    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.588215 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.588195    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hh7m4\" (UniqueName: \"kubernetes.io/projected/7b97fd34-f75d-41d4-aedb-4e18fff4a1fd-kube-api-access-hh7m4\") pod \"e2e-distinct-simulated-kserve-59975d69f9-sptfl\" (UID: \"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.670795 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.670698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:09.939656 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:09.939581    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"]
Sep 10 15:52:09.942323 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:52:09.942293    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b97fd34_f75d_41d4_aedb_4e18fff4a1fd.slice/crio-99f46566611fa1bb95125c48cc2a9fd7ae9c53140973ce02398a163e01286f1e WatchSource:0}: Error finding container 99f46566611fa1bb95125c48cc2a9fd7ae9c53140973ce02398a163e01286f1e: Status 404 returned error can't find the container with id 99f46566611fa1bb95125c48cc2a9fd7ae9c53140973ce02398a163e01286f1e
Sep 10 15:52:10.058473 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:10.058430    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:10.303221 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:10.303185    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl" event={"ID":"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd","Type":"ContainerStarted","Data":"99f46566611fa1bb95125c48cc2a9fd7ae9c53140973ce02398a163e01286f1e"}
Sep 10 15:52:11.308660 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:11.308624    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl" event={"ID":"7b97fd34-f75d-41d4-aedb-4e18fff4a1fd","Type":"ContainerStarted","Data":"490b5c19dc3b126081e5a7e7b1f83811459fa4315f8c1c3bfcaaab8c63ff0d16"}
Sep 10 15:52:11.326930 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:11.326861    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl" podStartSLOduration=1.338059629 podStartE2EDuration="2.32684119s" totalImagesPullingTime="988.781561ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:11.326021747 +0000 UTC m=+609.705968825" watchObservedRunningTime="2026-09-10 15:52:11.32684119 +0000 UTC m=+609.706788247"
Sep 10 15:52:17.853353 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:17.853306    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:19.671557 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:19.671506    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:19.671970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:19.671599    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:19.683689 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:19.683666    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:20.352848 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:20.352816    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-sptfl"
Sep 10 15:52:23.455872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:23.455795    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:26.951421 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:26.951382    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"]
Sep 10 15:52:26.954596 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:26.954571    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:26.957144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:26.957122    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 10 15:52:26.965259 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:26.965235    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"]
Sep 10 15:52:27.023178 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023145    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.023411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023185    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.023411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023214    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.023411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023276    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad76661f-6513-4aa4-852d-42b368b67519-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.023411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023304    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.023411 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.023328    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hpvgc\" (UniqueName: \"kubernetes.io/projected/ad76661f-6513-4aa4-852d-42b368b67519-kube-api-access-hpvgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124636 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124601    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124636 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124639    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124757    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad76661f-6513-4aa4-852d-42b368b67519-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124805    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124895 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124833    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hpvgc\" (UniqueName: \"kubernetes.io/projected/ad76661f-6513-4aa4-852d-42b368b67519-kube-api-access-hpvgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.124895 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.124862    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.125036 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.125014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.125092 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.125052    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.125145 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.125107    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.127048 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.127016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ad76661f-6513-4aa4-852d-42b368b67519-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.127334 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.127315    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ad76661f-6513-4aa4-852d-42b368b67519-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.132960 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.132933    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hpvgc\" (UniqueName: \"kubernetes.io/projected/ad76661f-6513-4aa4-852d-42b368b67519-kube-api-access-hpvgc\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn\" (UID: \"ad76661f-6513-4aa4-852d-42b368b67519\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.264900 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.264850    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:27.534486 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.534462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"]
Sep 10 15:52:27.536989 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:52:27.536959    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podad76661f_6513_4aa4_852d_42b368b67519.slice/crio-22c8a5dcffcb9a1973ed1aeadb7b190cfde3e95ccbdb8622c68fc39a4a84bc14 WatchSource:0}: Error finding container 22c8a5dcffcb9a1973ed1aeadb7b190cfde3e95ccbdb8622c68fc39a4a84bc14: Status 404 returned error can't find the container with id 22c8a5dcffcb9a1973ed1aeadb7b190cfde3e95ccbdb8622c68fc39a4a84bc14
Sep 10 15:52:27.538515 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:27.538494    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 15:52:28.252153 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:28.252117    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:52:28.377035 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:28.377003    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn" event={"ID":"ad76661f-6513-4aa4-852d-42b368b67519","Type":"ContainerStarted","Data":"8754a40892a887aa674de993a422e0f70120fa6da32598ec63c68ff0174e86c8"}
Sep 10 15:52:28.377210 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:28.377052    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn" event={"ID":"ad76661f-6513-4aa4-852d-42b368b67519","Type":"ContainerStarted","Data":"22c8a5dcffcb9a1973ed1aeadb7b190cfde3e95ccbdb8622c68fc39a4a84bc14"}
Sep 10 15:52:28.398859 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:28.398808    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn" podStartSLOduration=2.206411308 podStartE2EDuration="2.398792358s" totalImagesPullingTime="192.38105ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:52:26 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:52:28.397143141 +0000 UTC m=+626.777090220" watchObservedRunningTime="2026-09-10 15:52:28.398792358 +0000 UTC m=+626.778739423"
Sep 10 15:52:37.265081 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:37.265041    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:37.265501 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:37.265112    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:37.277484 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:37.277455    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:37.426200 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:37.426169    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn"
Sep 10 15:52:56.957770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:52:56.957732    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:53:14.746150 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.746116    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"]
Sep 10 15:53:14.749506 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.749479    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.752459 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.752433    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 10 15:53:14.764848 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.764811    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"]
Sep 10 15:53:14.786012 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.785979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.786181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.786023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.786181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.786080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.786181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.786138    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.786181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.786155    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3af9f0-fc25-4bba-b71f-d830336ec488-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.786181 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.786179    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfh6g\" (UniqueName: \"kubernetes.io/projected/3f3af9f0-fc25-4bba-b71f-d830336ec488-kube-api-access-pfh6g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887301 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887267    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887445 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887312    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887445 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887367    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887445 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887393    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3af9f0-fc25-4bba-b71f-d830336ec488-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887445 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887425    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfh6g\" (UniqueName: \"kubernetes.io/projected/3f3af9f0-fc25-4bba-b71f-d830336ec488-kube-api-access-pfh6g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887601 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887466    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887765 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887742    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887829 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887773    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.887880 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.887827    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.889879 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.889858    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3f3af9f0-fc25-4bba-b71f-d830336ec488-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.890008 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.889989    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3f3af9f0-fc25-4bba-b71f-d830336ec488-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:14.896214 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:14.896186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfh6g\" (UniqueName: \"kubernetes.io/projected/3f3af9f0-fc25-4bba-b71f-d830336ec488-kube-api-access-pfh6g\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc\" (UID: \"3f3af9f0-fc25-4bba-b71f-d830336ec488\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:15.059254 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:15.059149    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:15.331141 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:15.331116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"]
Sep 10 15:53:15.333672 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:53:15.333647    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3f3af9f0_fc25_4bba_b71f_d830336ec488.slice/crio-f8e26dc22f5b47c3083f679b25d273d546afac1919805ddcdfd6405fc79fb028 WatchSource:0}: Error finding container f8e26dc22f5b47c3083f679b25d273d546afac1919805ddcdfd6405fc79fb028: Status 404 returned error can't find the container with id f8e26dc22f5b47c3083f679b25d273d546afac1919805ddcdfd6405fc79fb028
Sep 10 15:53:15.570453 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:15.570417    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc" event={"ID":"3f3af9f0-fc25-4bba-b71f-d830336ec488","Type":"ContainerStarted","Data":"f8e26dc22f5b47c3083f679b25d273d546afac1919805ddcdfd6405fc79fb028"}
Sep 10 15:53:16.576484 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:16.576445    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc" event={"ID":"3f3af9f0-fc25-4bba-b71f-d830336ec488","Type":"ContainerStarted","Data":"5c0106f05e38e6a872bd1eccdd48bec116fab03b8abca99866c8561f6d281490"}
Sep 10 15:53:16.596655 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:16.596600    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc" podStartSLOduration=2.361979647 podStartE2EDuration="2.59658607s" totalImagesPullingTime="234.606423ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:53:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:53:16.595791435 +0000 UTC m=+674.975738540" watchObservedRunningTime="2026-09-10 15:53:16.59658607 +0000 UTC m=+674.976533130"
Sep 10 15:53:16.956845 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:16.956759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:53:25.059455 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:25.059410    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:25.059917 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:25.059478    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:25.071863 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:25.071835    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:53:25.624131 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:53:25.624104    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc"
Sep 10 15:54:05.817211 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.817136    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6675789b4c-x59x5"]
Sep 10 15:54:05.823790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.823761    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:05.827249 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.827225    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Sep 10 15:54:05.827961 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.827800    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-fswpm\""
Sep 10 15:54:05.828201 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.828181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6675789b4c-x59x5"]
Sep 10 15:54:05.998755 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.998720    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhs8k\" (UniqueName: \"kubernetes.io/projected/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-kube-api-access-nhs8k\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:05.998937 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:05.998812    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-tls-cert\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.100148 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.100051    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhs8k\" (UniqueName: \"kubernetes.io/projected/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-kube-api-access-nhs8k\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.100148 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.100141    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-tls-cert\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.102655 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.102637    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-tls-cert\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.108933 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.108911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhs8k\" (UniqueName: \"kubernetes.io/projected/fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd-kube-api-access-nhs8k\") pod \"authorino-6675789b4c-x59x5\" (UID: \"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd\") " pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.133189 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.133157    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6675789b4c-x59x5"
Sep 10 15:54:06.403491 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.403411    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6675789b4c-x59x5"]
Sep 10 15:54:06.406620 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:06.406588    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podfb8faa6f_e3d5_4a8d_be2b_99d4d6ece9fd.slice/crio-810bb4079734a68cfa67606e6c7b5a44b1807b127d7e31f57d8a1394e2e0c92d WatchSource:0}: Error finding container 810bb4079734a68cfa67606e6c7b5a44b1807b127d7e31f57d8a1394e2e0c92d: Status 404 returned error can't find the container with id 810bb4079734a68cfa67606e6c7b5a44b1807b127d7e31f57d8a1394e2e0c92d
Sep 10 15:54:06.786392 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:06.786334    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6675789b4c-x59x5" event={"ID":"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd","Type":"ContainerStarted","Data":"810bb4079734a68cfa67606e6c7b5a44b1807b127d7e31f57d8a1394e2e0c92d"}
Sep 10 15:54:07.793279 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:07.793181    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6675789b4c-x59x5" event={"ID":"fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd","Type":"ContainerStarted","Data":"18d36589411164f4f05c489fde100e128ddedf258dd0cb3fa5f1ab4b3fca79dd"}
Sep 10 15:54:07.812440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:07.812386    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-6675789b4c-x59x5" podStartSLOduration=1.723097187 podStartE2EDuration="2.812370634s" totalImagesPullingTime="1.089273447s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:07.811068589 +0000 UTC m=+726.191015661" watchObservedRunningTime="2026-09-10 15:54:07.812370634 +0000 UTC m=+726.192317699"
Sep 10 15:54:29.417176 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.417143    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7529-c9f77b"]
Sep 10 15:54:29.420392 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.420373    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:29.423005 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.422983    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c44xm\""
Sep 10 15:54:29.428414 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.428386    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-c9f77b"]
Sep 10 15:54:29.485774 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.485733    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r7x8w\" (UniqueName: \"kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w\") pod \"test-curl-7529-c9f77b\" (UID: \"2432637c-696a-4f0a-92d3-56c0b50d7dc3\") " pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:29.586958 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.586927    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r7x8w\" (UniqueName: \"kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w\") pod \"test-curl-7529-c9f77b\" (UID: \"2432637c-696a-4f0a-92d3-56c0b50d7dc3\") " pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:29.597073 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.597049    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r7x8w\" (UniqueName: \"kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w\") pod \"test-curl-7529-c9f77b\" (UID: \"2432637c-696a-4f0a-92d3-56c0b50d7dc3\") " pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:29.730731 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.730693    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:29.997925 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:29.997841    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-c9f77b"]
Sep 10 15:54:30.000967 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:30.000939    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod2432637c_696a_4f0a_92d3_56c0b50d7dc3.slice/crio-33cc7a0c8870a0c542bfcfecbdbfb6f1ed09f5478c48a11c0fa4f27773e7ff83 WatchSource:0}: Error finding container 33cc7a0c8870a0c542bfcfecbdbfb6f1ed09f5478c48a11c0fa4f27773e7ff83: Status 404 returned error can't find the container with id 33cc7a0c8870a0c542bfcfecbdbfb6f1ed09f5478c48a11c0fa4f27773e7ff83
Sep 10 15:54:30.899650 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:30.899612    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-c9f77b" event={"ID":"2432637c-696a-4f0a-92d3-56c0b50d7dc3","Type":"ContainerStarted","Data":"33cc7a0c8870a0c542bfcfecbdbfb6f1ed09f5478c48a11c0fa4f27773e7ff83"}
Sep 10 15:54:31.468276 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:31.468235    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:54:32.911067 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:32.911032    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-c9f77b" event={"ID":"2432637c-696a-4f0a-92d3-56c0b50d7dc3","Type":"ContainerStarted","Data":"a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5"}
Sep 10 15:54:32.927581 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:32.927529    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7529-c9f77b" podStartSLOduration=1.803684671 podStartE2EDuration="3.927511676s" totalImagesPullingTime="2.123827005s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:32.926559295 +0000 UTC m=+751.306506360" watchObservedRunningTime="2026-09-10 15:54:32.927511676 +0000 UTC m=+751.307458743"
Sep 10 15:54:33.547100 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.547065    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7529-c9f77b"]
Sep 10 15:54:33.553152 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.553110    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7529-c9f77b"]
Sep 10 15:54:33.690621 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.690549    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7529-22c643"]
Sep 10 15:54:33.690875 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.690863    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" containerName="test-curl-7529-c9f77b"
Sep 10 15:54:33.690917 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.690877    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" containerName="test-curl-7529-c9f77b"
Sep 10 15:54:33.690992 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.690968    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" containerName="test-curl-7529-c9f77b"
Sep 10 15:54:33.693847 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.693831    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:33.702385 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.702359    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-22c643"]
Sep 10 15:54:33.722731 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.722692    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sg7tz\" (UniqueName: \"kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz\") pod \"test-curl-7529-22c643\" (UID: \"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7\") " pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:33.824059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.824016    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sg7tz\" (UniqueName: \"kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz\") pod \"test-curl-7529-22c643\" (UID: \"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7\") " pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:33.833644 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:33.833620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sg7tz\" (UniqueName: \"kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz\") pod \"test-curl-7529-22c643\" (UID: \"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7\") " pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:33.915203 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:33.915171    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7529-c9f77b" container="test-curl-7529-c9f77b"
Sep 10 15:54:34.002425 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.002390    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:34.271595 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.271522    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7529-22c643"]
Sep 10 15:54:34.274482 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:34.274452    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1974dbfc_fb27_458a_b0a5_f319a3b1a6e7.slice/crio-98761a8cfef5c2b46381f171121aba3119a02097b3268a172a96d4839b96f133 WatchSource:0}: Error finding container 98761a8cfef5c2b46381f171121aba3119a02097b3268a172a96d4839b96f133: Status 404 returned error can't find the container with id 98761a8cfef5c2b46381f171121aba3119a02097b3268a172a96d4839b96f133
Sep 10 15:54:34.921089 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.921049    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-22c643" event={"ID":"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7","Type":"ContainerStarted","Data":"51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a"}
Sep 10 15:54:34.921089 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.921094    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7529-22c643" event={"ID":"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7","Type":"ContainerStarted","Data":"98761a8cfef5c2b46381f171121aba3119a02097b3268a172a96d4839b96f133"}
Sep 10 15:54:34.921663 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.921235    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7529-c9f77b" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" containerName="test-curl-7529-c9f77b" containerID="cri-o://a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5" gracePeriod=2
Sep 10 15:54:34.939967 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:34.939913    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7529-22c643" podStartSLOduration=1.939896966 podStartE2EDuration="1.939896966s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:34.938265973 +0000 UTC m=+753.318213040" watchObservedRunningTime="2026-09-10 15:54:34.939896966 +0000 UTC m=+753.319844031"
Sep 10 15:54:35.511169 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:35.511130    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7529-22c643"]
Sep 10 15:54:35.517000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:35.516966    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7529-22c643"]
Sep 10 15:54:35.925102 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:35.925073    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7529-22c643" container="test-curl-7529-22c643"
Sep 10 15:54:36.930091 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:36.930060    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7529-22c643" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" containerName="test-curl-7529-22c643" containerID="cri-o://51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a" gracePeriod=2
Sep 10 15:54:37.101970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.101943    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:37.149268 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.149236    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r7x8w\" (UniqueName: \"kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w\") pod \"2432637c-696a-4f0a-92d3-56c0b50d7dc3\" (UID: \"2432637c-696a-4f0a-92d3-56c0b50d7dc3\") "
Sep 10 15:54:37.151580 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.151557    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w" (OuterVolumeSpecName: "kube-api-access-r7x8w") pod "2432637c-696a-4f0a-92d3-56c0b50d7dc3" (UID: "2432637c-696a-4f0a-92d3-56c0b50d7dc3"). InnerVolumeSpecName "kube-api-access-r7x8w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:54:37.250838 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.250801    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r7x8w\" (UniqueName: \"kubernetes.io/projected/2432637c-696a-4f0a-92d3-56c0b50d7dc3-kube-api-access-r7x8w\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:54:37.934790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.934753    2560 generic.go:356] "Generic (PLEG): container finished" podID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" containerID="a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5" exitCode=137
Sep 10 15:54:37.934790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.934796    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:37.935247 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.934845    2560 scope.go:117] "RemoveContainer" containerID="a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5"
Sep 10 15:54:37.943180 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.943160    2560 scope.go:117] "RemoveContainer" containerID="a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5"
Sep 10 15:54:37.943467 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:37.943447    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5\": container with ID starting with a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5 not found: ID does not exist" containerID="a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5"
Sep 10 15:54:37.943520 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:37.943477    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5"} err="failed to get container status \"a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5\": rpc error: code = NotFound desc = could not find container \"a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5\": container with ID starting with a93a350a2ad2f71e8a3c356a500817ff5787d93a5511f1fcef5d644a903917a5 not found: ID does not exist"
Sep 10 15:54:37.945133 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:37.945110    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7529-c9f77b\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" pod="opendatahub/test-curl-7529-c9f77b"
Sep 10 15:54:38.115259 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:38.115222    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2432637c-696a-4f0a-92d3-56c0b50d7dc3" path="/var/lib/kubelet/pods/2432637c-696a-4f0a-92d3-56c0b50d7dc3/volumes"
Sep 10 15:54:39.111979 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.111955    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:39.165553 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.165521    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sg7tz\" (UniqueName: \"kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz\") pod \"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7\" (UID: \"1974dbfc-fb27-458a-b0a5-f319a3b1a6e7\") "
Sep 10 15:54:39.167737 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.167709    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz" (OuterVolumeSpecName: "kube-api-access-sg7tz") pod "1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" (UID: "1974dbfc-fb27-458a-b0a5-f319a3b1a6e7"). InnerVolumeSpecName "kube-api-access-sg7tz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:54:39.266767 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.266736    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sg7tz\" (UniqueName: \"kubernetes.io/projected/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7-kube-api-access-sg7tz\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:54:39.947402 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.947369    2560 generic.go:356] "Generic (PLEG): container finished" podID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" containerID="51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a" exitCode=137
Sep 10 15:54:39.947402 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.947396    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:39.947621 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.947451    2560 scope.go:117] "RemoveContainer" containerID="51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a"
Sep 10 15:54:39.956165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.956147    2560 scope.go:117] "RemoveContainer" containerID="51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a"
Sep 10 15:54:39.956474 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:39.956454    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a\": container with ID starting with 51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a not found: ID does not exist" containerID="51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a"
Sep 10 15:54:39.956533 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:39.956483    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a"} err="failed to get container status \"51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a\": rpc error: code = NotFound desc = could not find container \"51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a\": container with ID starting with 51d042d256bc31d7dad8adf1b6a18d431a840d9351643dcfa4f8f12a30bc035a not found: ID does not exist"
Sep 10 15:54:39.957293 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:54:39.957271    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7529-22c643\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" pod="opendatahub/test-curl-7529-22c643"
Sep 10 15:54:40.121044 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.121007    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" path="/var/lib/kubelet/pods/1974dbfc-fb27-458a-b0a5-f319a3b1a6e7/volumes"
Sep 10 15:54:40.774052 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.774015    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:54:40.774373 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.774358    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" containerName="test-curl-7529-22c643"
Sep 10 15:54:40.774424 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.774375    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" containerName="test-curl-7529-22c643"
Sep 10 15:54:40.774461 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.774430    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1974dbfc-fb27-458a-b0a5-f319a3b1a6e7" containerName="test-curl-7529-22c643"
Sep 10 15:54:40.778575 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.778558    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.780804 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.780781    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-54da81e9\""
Sep 10 15:54:40.780943 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.780864    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-54da81e9-dockercfg-wmtn4\""
Sep 10 15:54:40.788304 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.788281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:54:40.880244 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.880208    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkz26\" (UniqueName: \"kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.880417 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.880312    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.981356 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.981305    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.981552 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.981428    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkz26\" (UniqueName: \"kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.982066 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.982038    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:40.991111 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:40.991091    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkz26\" (UniqueName: \"kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26\") pod \"payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:41.088944 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.088864    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:41.360236 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.360163    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:54:41.363747 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:41.363704    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd51924c2_02bc_4748_86bc_415ca7ea9fe9.slice/crio-f875ca7f73a70b610aac060d52b5695f75296d5a6f83734d09155d0c87c0d4e0 WatchSource:0}: Error finding container f875ca7f73a70b610aac060d52b5695f75296d5a6f83734d09155d0c87c0d4e0: Status 404 returned error can't find the container with id f875ca7f73a70b610aac060d52b5695f75296d5a6f83734d09155d0c87c0d4e0
Sep 10 15:54:41.872171 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.872136    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:54:41.876526 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.876505    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:41.878890 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.878867    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-87a024\""
Sep 10 15:54:41.889872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.889832    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:54:41.958103 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.958061    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" event={"ID":"d51924c2-02bc-4748-86bc-415ca7ea9fe9","Type":"ContainerStarted","Data":"f875ca7f73a70b610aac060d52b5695f75296d5a6f83734d09155d0c87c0d4e0"}
Sep 10 15:54:41.989709 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.989668    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:41.989709 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:41.989710    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jbkrw\" (UniqueName: \"kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.091034 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.090992    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.091034 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.091035    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jbkrw\" (UniqueName: \"kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.091784 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.091762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.099521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.099495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jbkrw\" (UniqueName: \"kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw\") pod \"payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.186989 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.186905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:42.478752 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.478717    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:54:42.482546 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:42.482512    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89f926eb_e7c6_44c6_a877_ec7da6adc306.slice/crio-d493df4f48554834aa751ac80e31dbec35ab04274b3cbbdaf4e7064b19400e98 WatchSource:0}: Error finding container d493df4f48554834aa751ac80e31dbec35ab04274b3cbbdaf4e7064b19400e98: Status 404 returned error can't find the container with id d493df4f48554834aa751ac80e31dbec35ab04274b3cbbdaf4e7064b19400e98
Sep 10 15:54:42.963885 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:42.963810    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" event={"ID":"89f926eb-e7c6-44c6-a877-ec7da6adc306","Type":"ContainerStarted","Data":"d493df4f48554834aa751ac80e31dbec35ab04274b3cbbdaf4e7064b19400e98"}
Sep 10 15:54:44.973900 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:44.973854    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" event={"ID":"d51924c2-02bc-4748-86bc-415ca7ea9fe9","Type":"ContainerStarted","Data":"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"}
Sep 10 15:54:44.974362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:44.973958    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:44.975508 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:44.975482    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" event={"ID":"89f926eb-e7c6-44c6-a877-ec7da6adc306","Type":"ContainerStarted","Data":"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"}
Sep 10 15:54:44.975611 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:44.975547    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:44.989919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:44.989866    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" podStartSLOduration=2.443737245 podStartE2EDuration="4.989848913s" totalImagesPullingTime="2.546111668s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:40 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:44.988385694 +0000 UTC m=+763.368332761" watchObservedRunningTime="2026-09-10 15:54:44.989848913 +0000 UTC m=+763.369795970"
Sep 10 15:54:45.004124 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:45.004070    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" podStartSLOduration=2.576577209 podStartE2EDuration="4.004051927s" totalImagesPullingTime="1.427474718s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:45.002526083 +0000 UTC m=+763.382473149" watchObservedRunningTime="2026-09-10 15:54:45.004051927 +0000 UTC m=+763.383998986"
Sep 10 15:54:46.201456 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:46.201420    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:54:55.981782 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:55.981753    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:54:55.982219 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:55.981826    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:54:57.152948 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.152910    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:54:57.156056 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.156040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.158815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.158799    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-a-31b08b82-kserve-self-signed-certs\""
Sep 10 15:54:57.168565 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.168538    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:54:57.224009 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.223979    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.224009 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.224009    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.224315 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.224039    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.224315 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.224059    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.224315 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.224142    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.224315 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.224205    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s44rl\" (UniqueName: \"kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325482 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325441    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325642 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325492    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s44rl\" (UniqueName: \"kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325642 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325526    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325642 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325545    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325642 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325574    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325642 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325593    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.325930 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325906    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.326059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325929    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.326059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.325970    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.328053 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.328030    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.328205 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.328186    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.334217 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.334197    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s44rl\" (UniqueName: \"kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl\") pod \"e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") " pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.464783 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.464695    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:54:57.744157 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:57.744128    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:54:57.747365 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:54:57.747321    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode636ff7e_6fde_4f0b_870f_b0d33b8fb81e.slice/crio-df66eaab570f1f575a36ed923f1055c2431d6ff574a125e32631c1515828eb15 WatchSource:0}: Error finding container df66eaab570f1f575a36ed923f1055c2431d6ff574a125e32631c1515828eb15: Status 404 returned error can't find the container with id df66eaab570f1f575a36ed923f1055c2431d6ff574a125e32631c1515828eb15
Sep 10 15:54:58.031805 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:58.031708    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" event={"ID":"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e","Type":"ContainerStarted","Data":"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"}
Sep 10 15:54:58.031805 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:58.031747    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" event={"ID":"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e","Type":"ContainerStarted","Data":"df66eaab570f1f575a36ed923f1055c2431d6ff574a125e32631c1515828eb15"}
Sep 10 15:54:58.051792 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:54:58.051732    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" podStartSLOduration=1.051711804 podStartE2EDuration="1.051711804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:54:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:54:58.051046012 +0000 UTC m=+776.430993077" watchObservedRunningTime="2026-09-10 15:54:58.051711804 +0000 UTC m=+776.431658869"
Sep 10 15:55:02.588955 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.588921    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 15:55:02.593836 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.593815    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.595986 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.595962    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-607b9a\""
Sep 10 15:55:02.602551 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.602529    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 15:55:02.649164 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.649125    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 15:55:02.655034 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.655017    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.657597 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.657576    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-607b9a-dockercfg-v6jvl\""
Sep 10 15:55:02.663452 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.663429    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 15:55:02.673284 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.673258    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.673384 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.673291    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2kvzw\" (UniqueName: \"kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.774194 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.774158    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hf8hn\" (UniqueName: \"kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.774378 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.774236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.774378 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.774263    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.774378 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.774287    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2kvzw\" (UniqueName: \"kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.775018 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.774993    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.783509 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.783490    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2kvzw\" (UniqueName: \"kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw\") pod \"payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") " pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.874903 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.874827    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.874903 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.874880    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hf8hn\" (UniqueName: \"kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.875554 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.875533    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.884176 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.884147    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hf8hn\" (UniqueName: \"kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn\") pod \"payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:02.904466 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.904442    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:02.965836 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:02.965719    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:03.181702 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:03.181614    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 15:55:03.183798 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:55:03.183772    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b79d16b_46fc_41ea_a226_e3852c226887.slice/crio-0ec2fef0c6aa933bb468a4ed3e55a95e1a5365efae4fd892d0012385498cd03c WatchSource:0}: Error finding container 0ec2fef0c6aa933bb468a4ed3e55a95e1a5365efae4fd892d0012385498cd03c: Status 404 returned error can't find the container with id 0ec2fef0c6aa933bb468a4ed3e55a95e1a5365efae4fd892d0012385498cd03c
Sep 10 15:55:03.246837 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:03.246813    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 15:55:03.249916 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:55:03.249893    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1e0f9534_a706_48aa_a93e_33348ca7867d.slice/crio-dcd8f7fbbb61fdbfc07e7f3df3a6ffc34b46d2bc9205d6dc3a9fd6ac08093cc9 WatchSource:0}: Error finding container dcd8f7fbbb61fdbfc07e7f3df3a6ffc34b46d2bc9205d6dc3a9fd6ac08093cc9: Status 404 returned error can't find the container with id dcd8f7fbbb61fdbfc07e7f3df3a6ffc34b46d2bc9205d6dc3a9fd6ac08093cc9
Sep 10 15:55:04.058354 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.058312    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" event={"ID":"3b79d16b-46fc-41ea-a226-e3852c226887","Type":"ContainerStarted","Data":"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"}
Sep 10 15:55:04.058790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.058416    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" event={"ID":"3b79d16b-46fc-41ea-a226-e3852c226887","Type":"ContainerStarted","Data":"0ec2fef0c6aa933bb468a4ed3e55a95e1a5365efae4fd892d0012385498cd03c"}
Sep 10 15:55:04.058790 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.058445    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:04.059919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.059895    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" event={"ID":"1e0f9534-a706-48aa-a93e-33348ca7867d","Type":"ContainerStarted","Data":"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"}
Sep 10 15:55:04.060003 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.059935    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" event={"ID":"1e0f9534-a706-48aa-a93e-33348ca7867d","Type":"ContainerStarted","Data":"dcd8f7fbbb61fdbfc07e7f3df3a6ffc34b46d2bc9205d6dc3a9fd6ac08093cc9"}
Sep 10 15:55:04.060003 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.059981    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:04.078755 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.078703    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" podStartSLOduration=2.078688563 podStartE2EDuration="2.078688563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:04.076514612 +0000 UTC m=+782.456461670" watchObservedRunningTime="2026-09-10 15:55:04.078688563 +0000 UTC m=+782.458635629"
Sep 10 15:55:04.092905 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.092864    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" podStartSLOduration=2.092849135 podStartE2EDuration="2.092849135s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:04.092730756 +0000 UTC m=+782.472677823" watchObservedRunningTime="2026-09-10 15:55:04.092849135 +0000 UTC m=+782.472796206"
Sep 10 15:55:04.269759 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:04.269710    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:55:07.465847 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:07.465813    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:55:07.466335 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:07.465885    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:55:07.478558 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:07.478528    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:55:08.087289 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:08.087259    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:55:15.066311 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:15.066276    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 15:55:15.066739 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:15.066376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 15:55:19.363458 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:19.363425    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:55:21.090510 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.090476    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:55:21.090959 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.090694    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" podUID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" containerName="payload-pre-processing" containerID="cri-o://b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c" gracePeriod=30
Sep 10 15:55:21.340121 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.340092    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:55:21.434000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.433918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume\") pod \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") "
Sep 10 15:55:21.434000 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.433957    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bkz26\" (UniqueName: \"kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26\") pod \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\" (UID: \"d51924c2-02bc-4748-86bc-415ca7ea9fe9\") "
Sep 10 15:55:21.434383 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.434330    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d51924c2-02bc-4748-86bc-415ca7ea9fe9" (UID: "d51924c2-02bc-4748-86bc-415ca7ea9fe9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:21.436226 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.436208    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26" (OuterVolumeSpecName: "kube-api-access-bkz26") pod "d51924c2-02bc-4748-86bc-415ca7ea9fe9" (UID: "d51924c2-02bc-4748-86bc-415ca7ea9fe9"). InnerVolumeSpecName "kube-api-access-bkz26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:21.534809 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.534767    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d51924c2-02bc-4748-86bc-415ca7ea9fe9-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:21.534809 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:21.534802    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkz26\" (UniqueName: \"kubernetes.io/projected/d51924c2-02bc-4748-86bc-415ca7ea9fe9-kube-api-access-bkz26\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:22.134449 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.134417    2560 generic.go:356] "Generic (PLEG): container finished" podID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" containerID="b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c" exitCode=0
Sep 10 15:55:22.134801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.134459    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"
Sep 10 15:55:22.134801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.134493    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" event={"ID":"d51924c2-02bc-4748-86bc-415ca7ea9fe9","Type":"ContainerDied","Data":"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"}
Sep 10 15:55:22.134801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.134535    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8" event={"ID":"d51924c2-02bc-4748-86bc-415ca7ea9fe9","Type":"ContainerDied","Data":"f875ca7f73a70b610aac060d52b5695f75296d5a6f83734d09155d0c87c0d4e0"}
Sep 10 15:55:22.134801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.134558    2560 scope.go:117] "RemoveContainer" containerID="b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"
Sep 10 15:55:22.142671 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.142652    2560 scope.go:117] "RemoveContainer" containerID="b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"
Sep 10 15:55:22.142932 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:55:22.142907    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c\": container with ID starting with b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c not found: ID does not exist" containerID="b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"
Sep 10 15:55:22.143031 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.142938    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c"} err="failed to get container status \"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c\": rpc error: code = NotFound desc = could not find container \"b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c\": container with ID starting with b907cf4fcc62492a043ca908cd996cc8e8858ddfc69e4b42eed0ad0c2819e23c not found: ID does not exist"
Sep 10 15:55:22.151215 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.151192    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:55:22.157737 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:22.157716    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-54da81e9-75497c766b-z5gg8"]
Sep 10 15:55:24.115897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:24.115822    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" path="/var/lib/kubelet/pods/d51924c2-02bc-4748-86bc-415ca7ea9fe9/volumes"
Sep 10 15:55:26.674953 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.674915    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:26.675327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.675224    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" containerName="payload-pre-processing"
Sep 10 15:55:26.675327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.675235    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" containerName="payload-pre-processing"
Sep 10 15:55:26.675327 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.675295    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d51924c2-02bc-4748-86bc-415ca7ea9fe9" containerName="payload-pre-processing"
Sep 10 15:55:26.678127 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.678109    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.682273 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.682252    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-384e8433-dockercfg-n2j7m\""
Sep 10 15:55:26.682722 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.682693    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-384e8433\""
Sep 10 15:55:26.702070 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.702045    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:26.708561 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.708537    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:26.711548 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.711530    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.735919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.735898    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:26.771832 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.771803    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.771970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.771838    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.771970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.771891    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lm4cn\" (UniqueName: \"kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.771970 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.771952    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dqx6p\" (UniqueName: \"kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.872705 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.872657    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dqx6p\" (UniqueName: \"kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.872891 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.872734    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.872891 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.872768    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.872891 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.872817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lm4cn\" (UniqueName: \"kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.873475 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.873451    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.873585 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.873524    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.884301 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.884273    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lm4cn\" (UniqueName: \"kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn\") pod \"payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:26.884782 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.884762    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dqx6p\" (UniqueName: \"kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p\") pod \"payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") " pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:26.988369 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:26.988306    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:27.021949 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:27.021916    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:27.269901 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:27.269878    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:27.272995 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:55:27.272965    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcbf4c6a9_a22c_41dc_a64b_be1c9b274bee.slice/crio-35dc517bd1ccb72ac3ac0f87db852ede3758b7e70c1925672359bef7ca3e4234 WatchSource:0}: Error finding container 35dc517bd1ccb72ac3ac0f87db852ede3758b7e70c1925672359bef7ca3e4234: Status 404 returned error can't find the container with id 35dc517bd1ccb72ac3ac0f87db852ede3758b7e70c1925672359bef7ca3e4234
Sep 10 15:55:27.314214 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:27.314185    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:27.317408 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:55:27.317378    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ad85e46_e3e4_4fc9_9a26_eaf4e0ffd78e.slice/crio-5789968b18c78c3654eba39f306c837ead0150325e856eb623cee1fbb9857b09 WatchSource:0}: Error finding container 5789968b18c78c3654eba39f306c837ead0150325e856eb623cee1fbb9857b09: Status 404 returned error can't find the container with id 5789968b18c78c3654eba39f306c837ead0150325e856eb623cee1fbb9857b09
Sep 10 15:55:28.160959 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.160918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" event={"ID":"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e","Type":"ContainerStarted","Data":"32bf808035f91b54d19de18b4511c6888094dc846e4ef5989d184be9e1381144"}
Sep 10 15:55:28.161449 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.160972    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" event={"ID":"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e","Type":"ContainerStarted","Data":"5789968b18c78c3654eba39f306c837ead0150325e856eb623cee1fbb9857b09"}
Sep 10 15:55:28.161449 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.161045    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:28.162614 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.162591    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" event={"ID":"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee","Type":"ContainerStarted","Data":"810553ec7f5de3a2f8c1b1c6e26d20be97745191884154bb2dd13bb0e7f05b38"}
Sep 10 15:55:28.162681 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.162639    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" event={"ID":"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee","Type":"ContainerStarted","Data":"35dc517bd1ccb72ac3ac0f87db852ede3758b7e70c1925672359bef7ca3e4234"}
Sep 10 15:55:28.162681 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.162660    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:28.183144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.183099    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" podStartSLOduration=2.183085713 podStartE2EDuration="2.183085713s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:28.181368887 +0000 UTC m=+806.561315952" watchObservedRunningTime="2026-09-10 15:55:28.183085713 +0000 UTC m=+806.563032779"
Sep 10 15:55:28.207059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:28.206416    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" podStartSLOduration=2.206324561 podStartE2EDuration="2.206324561s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:55:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:55:28.206132384 +0000 UTC m=+806.586079445" watchObservedRunningTime="2026-09-10 15:55:28.206324561 +0000 UTC m=+806.586271629"
Sep 10 15:55:36.339683 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:36.339645    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:55:39.170323 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:39.170295    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:39.170887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:39.170483    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:57.167686 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.167651    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:57.168154 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.167876    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" podUID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" containerName="payload-processing" containerID="cri-o://32bf808035f91b54d19de18b4511c6888094dc846e4ef5989d184be9e1381144" gracePeriod=30
Sep 10 15:55:57.172724 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.172692    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:57.173143 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.173097    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" podUID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" containerName="payload-pre-processing" containerID="cri-o://810553ec7f5de3a2f8c1b1c6e26d20be97745191884154bb2dd13bb0e7f05b38" gracePeriod=30
Sep 10 15:55:57.290620 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.290592    2560 generic.go:356] "Generic (PLEG): container finished" podID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" containerID="32bf808035f91b54d19de18b4511c6888094dc846e4ef5989d184be9e1381144" exitCode=0
Sep 10 15:55:57.290760 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.290676    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" event={"ID":"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e","Type":"ContainerDied","Data":"32bf808035f91b54d19de18b4511c6888094dc846e4ef5989d184be9e1381144"}
Sep 10 15:55:57.292614 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.292591    2560 generic.go:356] "Generic (PLEG): container finished" podID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" containerID="810553ec7f5de3a2f8c1b1c6e26d20be97745191884154bb2dd13bb0e7f05b38" exitCode=0
Sep 10 15:55:57.292714 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.292625    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" event={"ID":"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee","Type":"ContainerDied","Data":"810553ec7f5de3a2f8c1b1c6e26d20be97745191884154bb2dd13bb0e7f05b38"}
Sep 10 15:55:57.419785 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.419708    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:57.449789 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.449764    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:57.537060 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537032    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lm4cn\" (UniqueName: \"kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn\") pod \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") "
Sep 10 15:55:57.537260 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537074    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume\") pod \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\" (UID: \"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee\") "
Sep 10 15:55:57.537260 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537123    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dqx6p\" (UniqueName: \"kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p\") pod \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") "
Sep 10 15:55:57.537260 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537139    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume\") pod \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\" (UID: \"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e\") "
Sep 10 15:55:57.537526 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537504    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" (UID: "cbf4c6a9-a22c-41dc-a64b-be1c9b274bee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:57.537646 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.537623    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" (UID: "0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:55:57.539524 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.539489    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn" (OuterVolumeSpecName: "kube-api-access-lm4cn") pod "cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" (UID: "cbf4c6a9-a22c-41dc-a64b-be1c9b274bee"). InnerVolumeSpecName "kube-api-access-lm4cn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:57.540973 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.540945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p" (OuterVolumeSpecName: "kube-api-access-dqx6p") pod "0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" (UID: "0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e"). InnerVolumeSpecName "kube-api-access-dqx6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:55:57.638267 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.638225    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lm4cn\" (UniqueName: \"kubernetes.io/projected/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-kube-api-access-lm4cn\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:57.638267 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.638261    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:57.638267 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.638271    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dqx6p\" (UniqueName: \"kubernetes.io/projected/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-kube-api-access-dqx6p\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:57.638528 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:57.638282    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:55:58.298236 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.298208    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"
Sep 10 15:55:58.298690 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.298206    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9" event={"ID":"0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e","Type":"ContainerDied","Data":"5789968b18c78c3654eba39f306c837ead0150325e856eb623cee1fbb9857b09"}
Sep 10 15:55:58.298690 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.298354    2560 scope.go:117] "RemoveContainer" containerID="32bf808035f91b54d19de18b4511c6888094dc846e4ef5989d184be9e1381144"
Sep 10 15:55:58.299749 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.299661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf" event={"ID":"cbf4c6a9-a22c-41dc-a64b-be1c9b274bee","Type":"ContainerDied","Data":"35dc517bd1ccb72ac3ac0f87db852ede3758b7e70c1925672359bef7ca3e4234"}
Sep 10 15:55:58.299749 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.299697    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"
Sep 10 15:55:58.307886 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.307868    2560 scope.go:117] "RemoveContainer" containerID="810553ec7f5de3a2f8c1b1c6e26d20be97745191884154bb2dd13bb0e7f05b38"
Sep 10 15:55:58.319276 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.319253    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:58.325736 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.325712    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-384e8433-586fc94f46-kpmf9"]
Sep 10 15:55:58.341489 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.341427    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:58.344977 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:58.344952    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-384e8433-86cbf8669d-hrgjf"]
Sep 10 15:55:59.455200 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:55:59.455160    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:56:00.116053 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:00.116017    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" path="/var/lib/kubelet/pods/0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e/volumes"
Sep 10 15:56:00.116386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:00.116374    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" path="/var/lib/kubelet/pods/cbf4c6a9-a22c-41dc-a64b-be1c9b274bee/volumes"
Sep 10 15:56:03.226440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226403    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:03.226821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226745    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" containerName="payload-pre-processing"
Sep 10 15:56:03.226821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226757    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" containerName="payload-pre-processing"
Sep 10 15:56:03.226821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226773    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" containerName="payload-processing"
Sep 10 15:56:03.226821 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226779    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" containerName="payload-processing"
Sep 10 15:56:03.226948 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226832    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ad85e46-e3e4-4fc9-9a26-eaf4e0ffd78e" containerName="payload-processing"
Sep 10 15:56:03.226948 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.226841    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="cbf4c6a9-a22c-41dc-a64b-be1c9b274bee" containerName="payload-pre-processing"
Sep 10 15:56:03.231180 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.231163    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.234955 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.234925    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-2dacd235-dockercfg-wclhs\""
Sep 10 15:56:03.234955 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.234928    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-2dacd235\""
Sep 10 15:56:03.248965 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.248928    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:03.318784 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.318741    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:03.322083 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.322059    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.333924 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.333899    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:03.388697 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.388662    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.388851 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.388714    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vs2ll\" (UniqueName: \"kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.489803 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.489707    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.489803 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.489755    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nmrss\" (UniqueName: \"kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.490002 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.489841    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.490002 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.489895    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vs2ll\" (UniqueName: \"kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.490551 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.490532    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.500367 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.500326    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vs2ll\" (UniqueName: \"kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll\") pod \"payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.541055 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.541018    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:03.592183 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.591537    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.592183 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.591732    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nmrss\" (UniqueName: \"kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.592741 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.592507    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.602554 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.602527    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nmrss\" (UniqueName: \"kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss\") pod \"payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") " pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.632424 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.632388    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:03.816645 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.816555    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:03.820520 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:56:03.820493    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c15b7c6_dac7_4cc4_bd48_5f574e7e5cb3.slice/crio-043a0648f504a72f5942011167bb61acacd52e1c10d44985ff55606d2de28d44 WatchSource:0}: Error finding container 043a0648f504a72f5942011167bb61acacd52e1c10d44985ff55606d2de28d44: Status 404 returned error can't find the container with id 043a0648f504a72f5942011167bb61acacd52e1c10d44985ff55606d2de28d44
Sep 10 15:56:03.912449 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:03.912414    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:03.915427 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:56:03.915402    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode30b2dbf_b059_4ebb_8816_ef0b90ec58ca.slice/crio-9ae7b9f8ff364f6a3a42772b64d79b56c172523e915ef88b2f7157ba6b1ee194 WatchSource:0}: Error finding container 9ae7b9f8ff364f6a3a42772b64d79b56c172523e915ef88b2f7157ba6b1ee194: Status 404 returned error can't find the container with id 9ae7b9f8ff364f6a3a42772b64d79b56c172523e915ef88b2f7157ba6b1ee194
Sep 10 15:56:04.331659 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.331624    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" event={"ID":"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3","Type":"ContainerStarted","Data":"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"}
Sep 10 15:56:04.332147 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.331692    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:04.332147 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.331714    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" event={"ID":"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3","Type":"ContainerStarted","Data":"043a0648f504a72f5942011167bb61acacd52e1c10d44985ff55606d2de28d44"}
Sep 10 15:56:04.333409 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.333375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" event={"ID":"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca","Type":"ContainerStarted","Data":"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"}
Sep 10 15:56:04.333540 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.333430    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" event={"ID":"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca","Type":"ContainerStarted","Data":"9ae7b9f8ff364f6a3a42772b64d79b56c172523e915ef88b2f7157ba6b1ee194"}
Sep 10 15:56:04.333540 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.333457    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:04.356148 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.356092    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" podStartSLOduration=1.356075594 podStartE2EDuration="1.356075594s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:04.355017008 +0000 UTC m=+842.734964074" watchObservedRunningTime="2026-09-10 15:56:04.356075594 +0000 UTC m=+842.736022659"
Sep 10 15:56:04.376387 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.376309    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" podStartSLOduration=1.376292877 podStartE2EDuration="1.376292877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:04.375898969 +0000 UTC m=+842.755846047" watchObservedRunningTime="2026-09-10 15:56:04.376292877 +0000 UTC m=+842.756239983"
Sep 10 15:56:04.757814 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.757779    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:56:04.761151 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.761125    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.763650 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.763630    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-conflict-b-31b08b82-kserve-self-signed-certs\""
Sep 10 15:56:04.774566 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.774542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:56:04.903770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.903725    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.903928 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.903793    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.903928 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.903868    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.903928 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.903913    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pl6v6\" (UniqueName: \"kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.904041 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.903975    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:04.904041 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:04.904012    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.004876 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.004840    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.004876 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.004884    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005106 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.004919    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005106 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.004946    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005106 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.004988    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005106 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.005036    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pl6v6\" (UniqueName: \"kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005361 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.005311    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005427 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.005367    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.005427 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.005418    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.007407 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.007390    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.007673 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.007656    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.014844 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.014772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pl6v6\" (UniqueName: \"kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6\") pod \"e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") " pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.070886 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.070855    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:05.345274 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:05.345199    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:56:05.347886 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:56:05.347859    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode14bcacd_7c7c_4fa5_b6a4_bed2593eb842.slice/crio-70004888a3ed728bafba8d3da43eb55c1dbd1cc046c164a91552937cbf412ae4 WatchSource:0}: Error finding container 70004888a3ed728bafba8d3da43eb55c1dbd1cc046c164a91552937cbf412ae4: Status 404 returned error can't find the container with id 70004888a3ed728bafba8d3da43eb55c1dbd1cc046c164a91552937cbf412ae4
Sep 10 15:56:06.342731 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:06.342679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" event={"ID":"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842","Type":"ContainerStarted","Data":"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"}
Sep 10 15:56:06.342907 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:06.342738    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" event={"ID":"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842","Type":"ContainerStarted","Data":"70004888a3ed728bafba8d3da43eb55c1dbd1cc046c164a91552937cbf412ae4"}
Sep 10 15:56:15.071551 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.071506    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:15.072156 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.071565    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:15.084167 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.084139    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:15.125602 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.125545    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" podStartSLOduration=11.125528661 podStartE2EDuration="11.125528661s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:06.366519313 +0000 UTC m=+844.746466378" watchObservedRunningTime="2026-09-10 15:56:15.125528661 +0000 UTC m=+853.505475754"
Sep 10 15:56:15.339887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.339805    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:15.339887 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.339880    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:15.393681 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:15.393647    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:56:18.802039 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:18.801995    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:56:33.373602 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.373565    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:33.374002 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.373833    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" podUID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" containerName="payload-processing" containerID="cri-o://d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022" gracePeriod=30
Sep 10 15:56:33.380745 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.380686    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:33.380934 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.380904    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" podUID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" containerName="payload-pre-processing" containerID="cri-o://7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6" gracePeriod=30
Sep 10 15:56:33.652309 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.652277    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:33.655160 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.655140    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:33.744230 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744184    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume\") pod \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") "
Sep 10 15:56:33.744436 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744259    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vs2ll\" (UniqueName: \"kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll\") pod \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") "
Sep 10 15:56:33.744436 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744298    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume\") pod \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\" (UID: \"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3\") "
Sep 10 15:56:33.744436 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744335    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nmrss\" (UniqueName: \"kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss\") pod \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\" (UID: \"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca\") "
Sep 10 15:56:33.744697 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744673    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" (UID: "e30b2dbf-b059-4ebb-8816-ef0b90ec58ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:33.744747 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.744704    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" (UID: "8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:56:33.746562 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.746542    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll" (OuterVolumeSpecName: "kube-api-access-vs2ll") pod "8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" (UID: "8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3"). InnerVolumeSpecName "kube-api-access-vs2ll". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:33.746671 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.746651    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss" (OuterVolumeSpecName: "kube-api-access-nmrss") pod "e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" (UID: "e30b2dbf-b059-4ebb-8816-ef0b90ec58ca"). InnerVolumeSpecName "kube-api-access-nmrss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:56:33.845412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.845375    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:56:33.845412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.845404    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vs2ll\" (UniqueName: \"kubernetes.io/projected/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-kube-api-access-vs2ll\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:56:33.845412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.845414    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:56:33.845412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:33.845422    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nmrss\" (UniqueName: \"kubernetes.io/projected/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca-kube-api-access-nmrss\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:56:34.464221 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.464128    2560 generic.go:356] "Generic (PLEG): container finished" podID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" containerID="7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6" exitCode=0
Sep 10 15:56:34.464221 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.464168    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"
Sep 10 15:56:34.464221 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.464201    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" event={"ID":"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3","Type":"ContainerDied","Data":"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"}
Sep 10 15:56:34.464748 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.464259    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf" event={"ID":"8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3","Type":"ContainerDied","Data":"043a0648f504a72f5942011167bb61acacd52e1c10d44985ff55606d2de28d44"}
Sep 10 15:56:34.464748 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.464291    2560 scope.go:117] "RemoveContainer" containerID="7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"
Sep 10 15:56:34.465670 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.465582    2560 generic.go:356] "Generic (PLEG): container finished" podID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" containerID="d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022" exitCode=0
Sep 10 15:56:34.465670 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.465627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"
Sep 10 15:56:34.465808 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.465680    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" event={"ID":"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca","Type":"ContainerDied","Data":"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"}
Sep 10 15:56:34.465808 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.465720    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz" event={"ID":"e30b2dbf-b059-4ebb-8816-ef0b90ec58ca","Type":"ContainerDied","Data":"9ae7b9f8ff364f6a3a42772b64d79b56c172523e915ef88b2f7157ba6b1ee194"}
Sep 10 15:56:34.472864 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.472845    2560 scope.go:117] "RemoveContainer" containerID="7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"
Sep 10 15:56:34.473165 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:56:34.473147    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6\": container with ID starting with 7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6 not found: ID does not exist" containerID="7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"
Sep 10 15:56:34.473204 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.473173    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6"} err="failed to get container status \"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6\": rpc error: code = NotFound desc = could not find container \"7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6\": container with ID starting with 7c391a62555d5d26e77a331163b5a55e0bdd924478875f16c78e9d5b503b4ef6 not found: ID does not exist"
Sep 10 15:56:34.473204 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.473189    2560 scope.go:117] "RemoveContainer" containerID="d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"
Sep 10 15:56:34.480663 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.480647    2560 scope.go:117] "RemoveContainer" containerID="d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"
Sep 10 15:56:34.480940 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:56:34.480907    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022\": container with ID starting with d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022 not found: ID does not exist" containerID="d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"
Sep 10 15:56:34.480940 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.480930    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022"} err="failed to get container status \"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022\": rpc error: code = NotFound desc = could not find container \"d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022\": container with ID starting with d5e0b3429abfae5a3c68e1d2561cdc70539eeb00da2ec98a18b9fff16d3da022 not found: ID does not exist"
Sep 10 15:56:34.487466 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.484796    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:34.489924 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.489902    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-2dacd235-7cf464bfd7-48xdz"]
Sep 10 15:56:34.503529 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.503503    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:34.508124 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:34.508102    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-2dacd235-6cdf75d74b-l7ctf"]
Sep 10 15:56:36.117961 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:36.117918    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" path="/var/lib/kubelet/pods/8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3/volumes"
Sep 10 15:56:36.118463 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:36.118449    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" path="/var/lib/kubelet/pods/e30b2dbf-b059-4ebb-8816-ef0b90ec58ca/volumes"
Sep 10 15:56:37.531838 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:37.531799    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:56:41.760754 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.760717    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:56:41.761129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761048    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" containerName="payload-pre-processing"
Sep 10 15:56:41.761129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761064    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" containerName="payload-pre-processing"
Sep 10 15:56:41.761129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761086    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" containerName="payload-processing"
Sep 10 15:56:41.761129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761092    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" containerName="payload-processing"
Sep 10 15:56:41.761256 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761166    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e30b2dbf-b059-4ebb-8816-ef0b90ec58ca" containerName="payload-processing"
Sep 10 15:56:41.761256 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.761174    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="8c15b7c6-dac7-4cc4-bd48-5f574e7e5cb3" containerName="payload-pre-processing"
Sep 10 15:56:41.765190 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.765172    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.772703 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.772684    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-8d528235-dockercfg-bhtml\""
Sep 10 15:56:41.780958 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.780933    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-8d528235\""
Sep 10 15:56:41.813856 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.813825    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.813998 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.813876    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8zwwk\" (UniqueName: \"kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.887864 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.887830    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:56:41.914574 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.914529    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8zwwk\" (UniqueName: \"kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.914763 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.914684    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.915538 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.915517    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:41.950874 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:41.950838    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8zwwk\" (UniqueName: \"kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk\") pod \"payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") " pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:42.075537 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.075429    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:42.369678 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.369651    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:56:42.371689 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:56:42.371649    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce48892a_d044_42c0_8150_ee7d86603a6f.slice/crio-01974a1c912501e27e5f36f80a972af36e32a363d5a3cc181331b1ad687b51a4 WatchSource:0}: Error finding container 01974a1c912501e27e5f36f80a972af36e32a363d5a3cc181331b1ad687b51a4: Status 404 returned error can't find the container with id 01974a1c912501e27e5f36f80a972af36e32a363d5a3cc181331b1ad687b51a4
Sep 10 15:56:42.505043 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.505002    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" event={"ID":"ce48892a-d044-42c0-8150-ee7d86603a6f","Type":"ContainerStarted","Data":"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"}
Sep 10 15:56:42.505203 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.505074    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" event={"ID":"ce48892a-d044-42c0-8150-ee7d86603a6f","Type":"ContainerStarted","Data":"01974a1c912501e27e5f36f80a972af36e32a363d5a3cc181331b1ad687b51a4"}
Sep 10 15:56:42.505203 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.505149    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:42.530266 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:42.530210    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" podStartSLOduration=1.53019155 podStartE2EDuration="1.53019155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:56:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:56:42.526114162 +0000 UTC m=+880.906061227" watchObservedRunningTime="2026-09-10 15:56:42.53019155 +0000 UTC m=+880.910138615"
Sep 10 15:56:53.510983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:53.510895    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:56:55.884770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:56:55.884730    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:57:10.875313 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:10.875279    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:57:11.566552 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:11.566520    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:57:15.928751 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:15.928707    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:57:21.308312 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.308274    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:57:21.310989 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.309069    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" podUID="ce48892a-d044-42c0-8150-ee7d86603a6f" containerName="payload-processing" containerID="cri-o://b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a" gracePeriod=30
Sep 10 15:57:21.572986 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.572928    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:57:21.674632 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.674596    2560 generic.go:356] "Generic (PLEG): container finished" podID="ce48892a-d044-42c0-8150-ee7d86603a6f" containerID="b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a" exitCode=0
Sep 10 15:57:21.674830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.674639    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"
Sep 10 15:57:21.674830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.674669    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" event={"ID":"ce48892a-d044-42c0-8150-ee7d86603a6f","Type":"ContainerDied","Data":"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"}
Sep 10 15:57:21.674830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.674739    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp" event={"ID":"ce48892a-d044-42c0-8150-ee7d86603a6f","Type":"ContainerDied","Data":"01974a1c912501e27e5f36f80a972af36e32a363d5a3cc181331b1ad687b51a4"}
Sep 10 15:57:21.674830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.674773    2560 scope.go:117] "RemoveContainer" containerID="b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"
Sep 10 15:57:21.679713 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.679686    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-8zwwk\" (UniqueName: \"kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk\") pod \"ce48892a-d044-42c0-8150-ee7d86603a6f\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") "
Sep 10 15:57:21.680481 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.679980    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume\") pod \"ce48892a-d044-42c0-8150-ee7d86603a6f\" (UID: \"ce48892a-d044-42c0-8150-ee7d86603a6f\") "
Sep 10 15:57:21.680803 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.680764    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ce48892a-d044-42c0-8150-ee7d86603a6f" (UID: "ce48892a-d044-42c0-8150-ee7d86603a6f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:21.682788 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.682753    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk" (OuterVolumeSpecName: "kube-api-access-8zwwk") pod "ce48892a-d044-42c0-8150-ee7d86603a6f" (UID: "ce48892a-d044-42c0-8150-ee7d86603a6f"). InnerVolumeSpecName "kube-api-access-8zwwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:21.685631 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.685598    2560 scope.go:117] "RemoveContainer" containerID="b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"
Sep 10 15:57:21.686009 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:57:21.685982    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a\": container with ID starting with b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a not found: ID does not exist" containerID="b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"
Sep 10 15:57:21.686095 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.686023    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a"} err="failed to get container status \"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a\": rpc error: code = NotFound desc = could not find container \"b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a\": container with ID starting with b94b475cc877bab8d6b5c4556f5028719195703da20a0c80eb94af65cb54054a not found: ID does not exist"
Sep 10 15:57:21.781216 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.781179    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ce48892a-d044-42c0-8150-ee7d86603a6f-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:21.781216 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:21.781210    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8zwwk\" (UniqueName: \"kubernetes.io/projected/ce48892a-d044-42c0-8150-ee7d86603a6f-kube-api-access-8zwwk\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:22.011886 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:22.011852    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:57:22.016316 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:22.016293    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-8d528235-8457b7ccbf-8qblp"]
Sep 10 15:57:22.115636 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:22.115602    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ce48892a-d044-42c0-8150-ee7d86603a6f" path="/var/lib/kubelet/pods/ce48892a-d044-42c0-8150-ee7d86603a6f/volumes"
Sep 10 15:57:24.749856 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:24.749814    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:57:24.750606 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:24.750067    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" podUID="89f926eb-e7c6-44c6-a877-ec7da6adc306" containerName="payload-pre-processing" containerID="cri-o://dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb" gracePeriod=30
Sep 10 15:57:25.011543 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.011483    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:57:25.109293 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.109257    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume\") pod \"89f926eb-e7c6-44c6-a877-ec7da6adc306\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") "
Sep 10 15:57:25.109525 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.109323    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jbkrw\" (UniqueName: \"kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw\") pod \"89f926eb-e7c6-44c6-a877-ec7da6adc306\" (UID: \"89f926eb-e7c6-44c6-a877-ec7da6adc306\") "
Sep 10 15:57:25.109718 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.109693    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "89f926eb-e7c6-44c6-a877-ec7da6adc306" (UID: "89f926eb-e7c6-44c6-a877-ec7da6adc306"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:57:25.111575 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.111550    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw" (OuterVolumeSpecName: "kube-api-access-jbkrw") pod "89f926eb-e7c6-44c6-a877-ec7da6adc306" (UID: "89f926eb-e7c6-44c6-a877-ec7da6adc306"). InnerVolumeSpecName "kube-api-access-jbkrw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:25.210119 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.210078    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/89f926eb-e7c6-44c6-a877-ec7da6adc306-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:25.210119 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.210113    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jbkrw\" (UniqueName: \"kubernetes.io/projected/89f926eb-e7c6-44c6-a877-ec7da6adc306-kube-api-access-jbkrw\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:25.694701 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.694669    2560 generic.go:356] "Generic (PLEG): container finished" podID="89f926eb-e7c6-44c6-a877-ec7da6adc306" containerID="dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb" exitCode=0
Sep 10 15:57:25.694896 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.694717    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"
Sep 10 15:57:25.694896 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.694763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" event={"ID":"89f926eb-e7c6-44c6-a877-ec7da6adc306","Type":"ContainerDied","Data":"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"}
Sep 10 15:57:25.694896 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.694813    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn" event={"ID":"89f926eb-e7c6-44c6-a877-ec7da6adc306","Type":"ContainerDied","Data":"d493df4f48554834aa751ac80e31dbec35ab04274b3cbbdaf4e7064b19400e98"}
Sep 10 15:57:25.694896 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.694836    2560 scope.go:117] "RemoveContainer" containerID="dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"
Sep 10 15:57:25.703551 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.703533    2560 scope.go:117] "RemoveContainer" containerID="dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"
Sep 10 15:57:25.703808 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:57:25.703790    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb\": container with ID starting with dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb not found: ID does not exist" containerID="dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"
Sep 10 15:57:25.703859 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.703816    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb"} err="failed to get container status \"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb\": rpc error: code = NotFound desc = could not find container \"dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb\": container with ID starting with dfe30354b3cca228d5299e6ce831523753d0aa18af459218189db60191dfc0fb not found: ID does not exist"
Sep 10 15:57:25.718557 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.718532    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:57:25.720579 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.720558    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-87a024-845f79ccfd-9qczn"]
Sep 10 15:57:25.895075 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:25.895037    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" podUID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" containerName="main" containerID="cri-o://e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf" gracePeriod=45
Sep 10 15:57:26.118139 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.118105    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89f926eb-e7c6-44c6-a877-ec7da6adc306" path="/var/lib/kubelet/pods/89f926eb-e7c6-44c6-a877-ec7da6adc306/volumes"
Sep 10 15:57:26.140289 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.140267    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:57:26.219691 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219657    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.219918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219724    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s44rl\" (UniqueName: \"kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.219918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219796    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.219918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219834    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.219918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219882    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.219918 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.219907    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home\") pod \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\" (UID: \"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e\") "
Sep 10 15:57:26.220168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.220003    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache" (OuterVolumeSpecName: "model-cache") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.220168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.220070    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.220264 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.220178    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.220264 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.220200    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.220264 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.220238    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home" (OuterVolumeSpecName: "home") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.222179 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.222154    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:26.222277 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.222182    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm" (OuterVolumeSpecName: "dshm") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.222277 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.222190    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl" (OuterVolumeSpecName: "kube-api-access-s44rl") pod "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" (UID: "e636ff7e-6fde-4f0b-870f-b0d33b8fb81e"). InnerVolumeSpecName "kube-api-access-s44rl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:26.321270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.321190    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.321270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.321218    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.321270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.321227    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.321270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.321235    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s44rl\" (UniqueName: \"kubernetes.io/projected/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e-kube-api-access-s44rl\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:26.586137 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.586044    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" podUID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" containerName="main" containerID="cri-o://76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd" gracePeriod=45
Sep 10 15:57:26.700521 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.700488    2560 generic.go:356] "Generic (PLEG): container finished" podID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" containerID="e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf" exitCode=0
Sep 10 15:57:26.700707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.700547    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"
Sep 10 15:57:26.700707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.700569    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" event={"ID":"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e","Type":"ContainerDied","Data":"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"}
Sep 10 15:57:26.700707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.700619    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf" event={"ID":"e636ff7e-6fde-4f0b-870f-b0d33b8fb81e","Type":"ContainerDied","Data":"df66eaab570f1f575a36ed923f1055c2431d6ff574a125e32631c1515828eb15"}
Sep 10 15:57:26.700707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.700642    2560 scope.go:117] "RemoveContainer" containerID="e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"
Sep 10 15:57:26.709469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.709454    2560 scope.go:117] "RemoveContainer" containerID="e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"
Sep 10 15:57:26.709744 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:57:26.709721    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf\": container with ID starting with e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf not found: ID does not exist" containerID="e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"
Sep 10 15:57:26.709838 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.709748    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf"} err="failed to get container status \"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf\": rpc error: code = NotFound desc = could not find container \"e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf\": container with ID starting with e73a40a754cc8e7fc1380eff63d4aeaffe18ce547b8d5b86392fd369eaf170bf not found: ID does not exist"
Sep 10 15:57:26.731672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.731632    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:57:26.762017 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.761978    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-a-31b08b82-kserve-b54f6bbc-kvnwf"]
Sep 10 15:57:26.851575 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.851550    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:57:26.927207 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927179    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pl6v6\" (UniqueName: \"kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927568 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927214    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927568 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927276    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927568 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927293    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927568 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927568 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927354    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs\") pod \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\" (UID: \"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842\") "
Sep 10 15:57:26.927815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927603    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home" (OuterVolumeSpecName: "home") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.927815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927618    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.927815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.927633    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache" (OuterVolumeSpecName: "model-cache") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.929575 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.929548    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm" (OuterVolumeSpecName: "dshm") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:57:26.929672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.929549    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:57:26.929672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:26.929645    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6" (OuterVolumeSpecName: "kube-api-access-pl6v6") pod "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" (UID: "e14bcacd-7c7c-4fa5-b6a4-bed2593eb842"). InnerVolumeSpecName "kube-api-access-pl6v6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:57:27.028572 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028526    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.028572 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028560    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.028572 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028569    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pl6v6\" (UniqueName: \"kubernetes.io/projected/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-kube-api-access-pl6v6\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.028572 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028580    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.028830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028589    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.028830 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.028618    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:57:27.708044 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.708007    2560 generic.go:356] "Generic (PLEG): container finished" podID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" containerID="76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd" exitCode=0
Sep 10 15:57:27.708044 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.708045    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"
Sep 10 15:57:27.708253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.708046    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" event={"ID":"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842","Type":"ContainerDied","Data":"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"}
Sep 10 15:57:27.708253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.708161    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b" event={"ID":"e14bcacd-7c7c-4fa5-b6a4-bed2593eb842","Type":"ContainerDied","Data":"70004888a3ed728bafba8d3da43eb55c1dbd1cc046c164a91552937cbf412ae4"}
Sep 10 15:57:27.708253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.708180    2560 scope.go:117] "RemoveContainer" containerID="76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"
Sep 10 15:57:27.717019 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.717002    2560 scope.go:117] "RemoveContainer" containerID="76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"
Sep 10 15:57:27.717286 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:57:27.717269    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd\": container with ID starting with 76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd not found: ID does not exist" containerID="76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"
Sep 10 15:57:27.717365 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.717296    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd"} err="failed to get container status \"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd\": rpc error: code = NotFound desc = could not find container \"76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd\": container with ID starting with 76be41ef243efe90bac5b0b7cd71736807e907f676198c410d49c169820c06bd not found: ID does not exist"
Sep 10 15:57:27.729511 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.729485    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:57:27.736553 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:27.736532    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-conflict-b-31b08b82-kserve-84bb765c85-l2d5b"]
Sep 10 15:57:28.115571 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:28.115529    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" path="/var/lib/kubelet/pods/e14bcacd-7c7c-4fa5-b6a4-bed2593eb842/volumes"
Sep 10 15:57:28.116090 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:28.116069    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" path="/var/lib/kubelet/pods/e636ff7e-6fde-4f0b-870f-b0d33b8fb81e/volumes"
Sep 10 15:57:33.672706 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:33.672667    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:57:38.689848 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.689813    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690094    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ce48892a-d044-42c0-8150-ee7d86603a6f" containerName="payload-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690105    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ce48892a-d044-42c0-8150-ee7d86603a6f" containerName="payload-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690119    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" containerName="main"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690125    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" containerName="main"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690135    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" containerName="main"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690140    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" containerName="main"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690150    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89f926eb-e7c6-44c6-a877-ec7da6adc306" containerName="payload-pre-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690155    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="89f926eb-e7c6-44c6-a877-ec7da6adc306" containerName="payload-pre-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690211    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e636ff7e-6fde-4f0b-870f-b0d33b8fb81e" containerName="main"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690218    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ce48892a-d044-42c0-8150-ee7d86603a6f" containerName="payload-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690226    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="89f926eb-e7c6-44c6-a877-ec7da6adc306" containerName="payload-pre-processing"
Sep 10 15:57:38.690257 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.690232    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e14bcacd-7c7c-4fa5-b6a4-bed2593eb842" containerName="main"
Sep 10 15:57:38.694477 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.694460    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.697129 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.697104    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-61fba2\""
Sep 10 15:57:38.697254 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.697173    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-61fba2-dockercfg-zcz8c\""
Sep 10 15:57:38.706100 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.706077    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:57:38.822030 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.821994    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vvndg\" (UniqueName: \"kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.822201 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.822046    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.923015 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.922975    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vvndg\" (UniqueName: \"kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.923202 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.923033    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.923720 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.923702    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:38.936958 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:38.936921    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vvndg\" (UniqueName: \"kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg\") pod \"payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:39.004705 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.004671    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:39.312511 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.312426    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:57:39.315373 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:57:39.315330    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfd953bb7_ee06_4716_84c5_663222cb5751.slice/crio-4b47cec0756e36e791e2c8c6eaa8ace10018f1c4904a48368a8b829a5b294880 WatchSource:0}: Error finding container 4b47cec0756e36e791e2c8c6eaa8ace10018f1c4904a48368a8b829a5b294880: Status 404 returned error can't find the container with id 4b47cec0756e36e791e2c8c6eaa8ace10018f1c4904a48368a8b829a5b294880
Sep 10 15:57:39.762931 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.762901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" event={"ID":"fd953bb7-ee06-4716-84c5-663222cb5751","Type":"ContainerStarted","Data":"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"}
Sep 10 15:57:39.763356 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.762950    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" event={"ID":"fd953bb7-ee06-4716-84c5-663222cb5751","Type":"ContainerStarted","Data":"4b47cec0756e36e791e2c8c6eaa8ace10018f1c4904a48368a8b829a5b294880"}
Sep 10 15:57:39.763356 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.763014    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:39.787144 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:39.787091    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" podStartSLOduration=1.787072254 podStartE2EDuration="1.787072254s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:39.785730666 +0000 UTC m=+938.165677736" watchObservedRunningTime="2026-09-10 15:57:39.787072254 +0000 UTC m=+938.167019319"
Sep 10 15:57:50.770106 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:50.770064    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:57:51.263054 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.263014    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:57:51.266353 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.266315    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.269872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.269845    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-607b9a\"/\"openshift-service-ca.crt\""
Sep 10 15:57:51.269872 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.269845    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-607b9a\"/\"default-dockercfg-n6f65\""
Sep 10 15:57:51.270055 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.269891    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-607b9a\"/\"kube-root-ca.crt\""
Sep 10 15:57:51.270055 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.269904    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-607b9a\"/\"test-model-607b9a-kserve-self-signed-certs\""
Sep 10 15:57:51.276882 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.276860    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:57:51.435775 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.435775 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435787    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.436061 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435861    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x4dtw\" (UniqueName: \"kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.436061 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435910    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.436061 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.436061 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.435962    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537080 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.536984    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537080 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537080 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537048    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537080 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537084    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537457 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537109    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537457 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537133    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x4dtw\" (UniqueName: \"kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537558 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537519    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537558 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537540    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.537626 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.537571    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.539644 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.539620    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.539750 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.539718    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.547388 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.547363    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x4dtw\" (UniqueName: \"kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw\") pod \"test-model-607b9a-kserve-545879fd7f-7qzm7\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") " pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.575194 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.575168    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:57:51.851332 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:51.851251    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:57:51.854620 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:57:51.854589    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9f77d1a0_da27_456a_8938_31f9e719df47.slice/crio-170be3fd6c4998965a2b7ef06cc1029d2e09407ccaf0029793c4d6022b7e18a0 WatchSource:0}: Error finding container 170be3fd6c4998965a2b7ef06cc1029d2e09407ccaf0029793c4d6022b7e18a0: Status 404 returned error can't find the container with id 170be3fd6c4998965a2b7ef06cc1029d2e09407ccaf0029793c4d6022b7e18a0
Sep 10 15:57:52.771184 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:52.771147    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:57:52.819893 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:52.819856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" event={"ID":"9f77d1a0-da27-456a-8938-31f9e719df47","Type":"ContainerStarted","Data":"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"}
Sep 10 15:57:52.819893 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:52.819901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" event={"ID":"9f77d1a0-da27-456a-8938-31f9e719df47","Type":"ContainerStarted","Data":"170be3fd6c4998965a2b7ef06cc1029d2e09407ccaf0029793c4d6022b7e18a0"}
Sep 10 15:57:52.846028 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:57:52.845969    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" podStartSLOduration=1.8459491099999998 podStartE2EDuration="1.84594911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:57:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:57:52.843659902 +0000 UTC m=+951.223606968" watchObservedRunningTime="2026-09-10 15:57:52.84594911 +0000 UTC m=+951.225896176"
Sep 10 15:58:01.575639 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:01.575592    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:58:01.576172 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:01.575658    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:58:01.588362 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:01.588311    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:58:01.868756 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:01.868676    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:58:08.977066 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:08.977024    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:58:13.557026 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.556991    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:13.561501 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.561482    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.564216 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.564192    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-61fba2\"/\"openshift-service-ca.crt\""
Sep 10 15:58:13.565451 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.565426    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-61fba2\"/\"kube-root-ca.crt\""
Sep 10 15:58:13.565451 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.565437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-61fba2\"/\"explicit-model-61fba2-kserve-self-signed-certs\""
Sep 10 15:58:13.565616 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.565437    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-61fba2\"/\"default-dockercfg-pftdz\""
Sep 10 15:58:13.571026 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.570999    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:13.596547 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.596547 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596550    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.596743 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596604    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.596743 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596645    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.596743 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596698    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.596743 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.596740    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qms6p\" (UniqueName: \"kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697781 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697743    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697781 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697941 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697816    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qms6p\" (UniqueName: \"kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697941 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697847    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697941 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.697941 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.697889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.698229 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.698204    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.698274 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.698218    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.698274 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.698264    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.700291 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.700270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.700546 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.700530    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.707769 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.707736    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qms6p\" (UniqueName: \"kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p\") pod \"explicit-model-61fba2-kserve-68dc8b79c7-gmn7m\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") " pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:13.870876 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:13.870791    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:14.145627 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:14.145599    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:14.147562 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:58:14.147533    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podea63c7c2_e5fa_4b9e_a271_25f6ae93195a.slice/crio-3c5d6c09dc01ca11e82dd16dd0de1d68774bc139f0ace035833083cdb8580e14 WatchSource:0}: Error finding container 3c5d6c09dc01ca11e82dd16dd0de1d68774bc139f0ace035833083cdb8580e14: Status 404 returned error can't find the container with id 3c5d6c09dc01ca11e82dd16dd0de1d68774bc139f0ace035833083cdb8580e14
Sep 10 15:58:14.911797 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:14.911763    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" event={"ID":"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a","Type":"ContainerStarted","Data":"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"}
Sep 10 15:58:14.912184 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:14.911806    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" event={"ID":"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a","Type":"ContainerStarted","Data":"3c5d6c09dc01ca11e82dd16dd0de1d68774bc139f0ace035833083cdb8580e14"}
Sep 10 15:58:14.934864 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:14.934801    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" podStartSLOduration=1.9347866169999999 podStartE2EDuration="1.934786617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:58:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:58:14.933692052 +0000 UTC m=+973.313639119" watchObservedRunningTime="2026-09-10 15:58:14.934786617 +0000 UTC m=+973.314733682"
Sep 10 15:58:23.870994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:23.870899    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:23.870994 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:23.870953    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:23.883651 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:23.883625    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:23.966157 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:23.966128    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:28.164168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:28.164130    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:58:33.996902 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:33.996871    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:44.563793 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:44.563750    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:58:44.727975 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:44.727939    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:58:44.728220 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:44.728172    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" podUID="fd953bb7-ee06-4716-84c5-663222cb5751" containerName="payload-pre-processing" containerID="cri-o://8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e" gracePeriod=30
Sep 10 15:58:44.973004 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:44.972981    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:58:45.043315 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.043283    2560 generic.go:356] "Generic (PLEG): container finished" podID="fd953bb7-ee06-4716-84c5-663222cb5751" containerID="8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e" exitCode=0
Sep 10 15:58:45.043515 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.043322    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"
Sep 10 15:58:45.043515 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.043375    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" event={"ID":"fd953bb7-ee06-4716-84c5-663222cb5751","Type":"ContainerDied","Data":"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"}
Sep 10 15:58:45.043515 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.043435    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m" event={"ID":"fd953bb7-ee06-4716-84c5-663222cb5751","Type":"ContainerDied","Data":"4b47cec0756e36e791e2c8c6eaa8ace10018f1c4904a48368a8b829a5b294880"}
Sep 10 15:58:45.043515 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.043466    2560 scope.go:117] "RemoveContainer" containerID="8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"
Sep 10 15:58:45.052471 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.052428    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vvndg\" (UniqueName: \"kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg\") pod \"fd953bb7-ee06-4716-84c5-663222cb5751\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") "
Sep 10 15:58:45.052593 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.052488    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume\") pod \"fd953bb7-ee06-4716-84c5-663222cb5751\" (UID: \"fd953bb7-ee06-4716-84c5-663222cb5751\") "
Sep 10 15:58:45.053949 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.053044    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fd953bb7-ee06-4716-84c5-663222cb5751" (UID: "fd953bb7-ee06-4716-84c5-663222cb5751"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:58:45.053949 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.053244    2560 scope.go:117] "RemoveContainer" containerID="8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"
Sep 10 15:58:45.053949 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:58:45.053925    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e\": container with ID starting with 8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e not found: ID does not exist" containerID="8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"
Sep 10 15:58:45.054083 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.053960    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e"} err="failed to get container status \"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e\": rpc error: code = NotFound desc = could not find container \"8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e\": container with ID starting with 8efd0c29deeeeb7a2d2407c4c3f94dd41f7b33baf19942ab3b18bc0e340fb31e not found: ID does not exist"
Sep 10 15:58:45.055747 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.055723    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg" (OuterVolumeSpecName: "kube-api-access-vvndg") pod "fd953bb7-ee06-4716-84c5-663222cb5751" (UID: "fd953bb7-ee06-4716-84c5-663222cb5751"). InnerVolumeSpecName "kube-api-access-vvndg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:45.153253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.153181    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vvndg\" (UniqueName: \"kubernetes.io/projected/fd953bb7-ee06-4716-84c5-663222cb5751-kube-api-access-vvndg\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:45.153253 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.153206    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fd953bb7-ee06-4716-84c5-663222cb5751-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:45.365615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.365577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:58:45.369868 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:45.369844    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-61fba2-75c4c6c7fd-w2h4m"]
Sep 10 15:58:46.115503 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:46.115470    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fd953bb7-ee06-4716-84c5-663222cb5751" path="/var/lib/kubelet/pods/fd953bb7-ee06-4716-84c5-663222cb5751/volumes"
Sep 10 15:58:49.016983 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.016911    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" podUID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" containerName="main" containerID="cri-o://648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e" gracePeriod=45
Sep 10 15:58:49.261517 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.261492    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:49.390504 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390392    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390504 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390462    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390504 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390485    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390504 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390504    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390504 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390520    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390574    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qms6p\" (UniqueName: \"kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p\") pod \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\" (UID: \"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a\") "
Sep 10 15:58:49.390897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390695    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache" (OuterVolumeSpecName: "model-cache") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:49.390897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390802    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:49.390897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390826    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home" (OuterVolumeSpecName: "home") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:49.390897 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.390857    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:49.392913 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.392880    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm" (OuterVolumeSpecName: "dshm") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:58:49.393030 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.392961    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:58:49.393030 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.392959    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p" (OuterVolumeSpecName: "kube-api-access-qms6p") pod "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" (UID: "ea63c7c2-e5fa-4b9e-a271-25f6ae93195a"). InnerVolumeSpecName "kube-api-access-qms6p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:58:49.492094 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.492054    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:49.492094 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.492086    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:49.492094 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.492094    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:49.492094 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.492103    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:49.492398 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:49.492111    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qms6p\" (UniqueName: \"kubernetes.io/projected/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a-kube-api-access-qms6p\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:58:50.066934 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.066897    2560 generic.go:356] "Generic (PLEG): container finished" podID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" containerID="648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e" exitCode=0
Sep 10 15:58:50.066934 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.066943    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"
Sep 10 15:58:50.067432 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.066959    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" event={"ID":"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a","Type":"ContainerDied","Data":"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"}
Sep 10 15:58:50.067432 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.066999    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m" event={"ID":"ea63c7c2-e5fa-4b9e-a271-25f6ae93195a","Type":"ContainerDied","Data":"3c5d6c09dc01ca11e82dd16dd0de1d68774bc139f0ace035833083cdb8580e14"}
Sep 10 15:58:50.067432 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.067022    2560 scope.go:117] "RemoveContainer" containerID="648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"
Sep 10 15:58:50.075974 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.075955    2560 scope.go:117] "RemoveContainer" containerID="648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"
Sep 10 15:58:50.076257 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:58:50.076240    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e\": container with ID starting with 648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e not found: ID does not exist" containerID="648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"
Sep 10 15:58:50.076302 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.076265    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e"} err="failed to get container status \"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e\": rpc error: code = NotFound desc = could not find container \"648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e\": container with ID starting with 648b531ebb634bccfda0241aeb5736635f8a761a3cb292f6599bd35d617df41e not found: ID does not exist"
Sep 10 15:58:50.088387 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.088361    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:50.093525 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.093504    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-61fba2/explicit-model-61fba2-kserve-68dc8b79c7-gmn7m"]
Sep 10 15:58:50.116267 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:50.116240    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" path="/var/lib/kubelet/pods/ea63c7c2-e5fa-4b9e-a271-25f6ae93195a/volumes"
Sep 10 15:58:59.465555 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:58:59.465518    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:59:08.133714 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.133681    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:08.134168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134025    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fd953bb7-ee06-4716-84c5-663222cb5751" containerName="payload-pre-processing"
Sep 10 15:59:08.134168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134042    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="fd953bb7-ee06-4716-84c5-663222cb5751" containerName="payload-pre-processing"
Sep 10 15:59:08.134168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134089    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" containerName="main"
Sep 10 15:59:08.134168 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134099    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" containerName="main"
Sep 10 15:59:08.134317 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134192    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ea63c7c2-e5fa-4b9e-a271-25f6ae93195a" containerName="main"
Sep 10 15:59:08.134317 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.134202    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="fd953bb7-ee06-4716-84c5-663222cb5751" containerName="payload-pre-processing"
Sep 10 15:59:08.138690 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.138673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.141287 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.141271    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-9a70cb1a-dockercfg-k97gp\""
Sep 10 15:59:08.141369 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.141274    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-9a70cb1a\""
Sep 10 15:59:08.151915 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.151887    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:08.201589 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.201551    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:08.204914 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.204895    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.214497 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.214472    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:08.252415 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.252365    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2822\" (UniqueName: \"kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.252615 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.252497    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.353785 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.353735    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2822\" (UniqueName: \"kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.353968 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.353832    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ng8g9\" (UniqueName: \"kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.353968 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.353889    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.354039 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.353955    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.354628 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.354611    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.364104 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.364067    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2822\" (UniqueName: \"kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822\") pod \"payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") " pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.448824 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.448743    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:08.455097 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.455073    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ng8g9\" (UniqueName: \"kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.455203 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.455126    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.455825 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.455789    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.466035 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.466007    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ng8g9\" (UniqueName: \"kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9\") pod \"payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.515475 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.515452    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:08.721749 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:59:08.721712    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e40832e_c20b_4310_997c_3f16a552beb6.slice/crio-4d141056edc352cf142dbbfece9895559e9d14bb6b078c736f7a56ab0adf3b65 WatchSource:0}: Error finding container 4d141056edc352cf142dbbfece9895559e9d14bb6b078c736f7a56ab0adf3b65: Status 404 returned error can't find the container with id 4d141056edc352cf142dbbfece9895559e9d14bb6b078c736f7a56ab0adf3b65
Sep 10 15:59:08.722050 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.722033    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:08.794936 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:08.794852    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:08.797243 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:59:08.797213    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda5490dfe_7475_459c_8c03_417a753b2a19.slice/crio-7480cb6d4c178f7128a436af2906395901d4ed37b5f107c6c6642600c35890c3 WatchSource:0}: Error finding container 7480cb6d4c178f7128a436af2906395901d4ed37b5f107c6c6642600c35890c3: Status 404 returned error can't find the container with id 7480cb6d4c178f7128a436af2906395901d4ed37b5f107c6c6642600c35890c3
Sep 10 15:59:09.153926 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.153881    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" event={"ID":"4e40832e-c20b-4310-997c-3f16a552beb6","Type":"ContainerStarted","Data":"35bb262adc95baf4cd43031a4ab98ef2f0e35183e9f125f967ad2bb7765b25d5"}
Sep 10 15:59:09.154394 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.153941    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" event={"ID":"4e40832e-c20b-4310-997c-3f16a552beb6","Type":"ContainerStarted","Data":"4d141056edc352cf142dbbfece9895559e9d14bb6b078c736f7a56ab0adf3b65"}
Sep 10 15:59:09.154394 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.153969    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:09.155529 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.155505    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" event={"ID":"a5490dfe-7475-459c-8c03-417a753b2a19","Type":"ContainerStarted","Data":"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"}
Sep 10 15:59:09.155624 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.155554    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" event={"ID":"a5490dfe-7475-459c-8c03-417a753b2a19","Type":"ContainerStarted","Data":"7480cb6d4c178f7128a436af2906395901d4ed37b5f107c6c6642600c35890c3"}
Sep 10 15:59:09.155624 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.155585    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:09.174786 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.174738    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" podStartSLOduration=1.174721512 podStartE2EDuration="1.174721512s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:09.173881986 +0000 UTC m=+1027.553829065" watchObservedRunningTime="2026-09-10 15:59:09.174721512 +0000 UTC m=+1027.554668602"
Sep 10 15:59:09.194053 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:09.194005    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" podStartSLOduration=1.193991431 podStartE2EDuration="1.193991431s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:09.192961332 +0000 UTC m=+1027.572908418" watchObservedRunningTime="2026-09-10 15:59:09.193991431 +0000 UTC m=+1027.573938546"
Sep 10 15:59:15.965835 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:15.965796    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:59:20.161641 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:20.161604    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:20.162039 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:20.161696    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:26.371321 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.371280    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 15:59:26.374685 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.374661    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.378335 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.378313    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-17d9a5\"/\"noten-model-17d9a5-kserve-self-signed-certs\""
Sep 10 15:59:26.378481 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.378354    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-17d9a5\"/\"openshift-service-ca.crt\""
Sep 10 15:59:26.378481 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.378370    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-noten-17d9a5\"/\"kube-root-ca.crt\""
Sep 10 15:59:26.378481 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.378445    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-noten-17d9a5\"/\"default-dockercfg-gjfrc\""
Sep 10 15:59:26.386725 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.386702    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 15:59:26.398518 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398488    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dps8s\" (UniqueName: \"kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.398640 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398534    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.398640 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398600    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.398770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398673    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.398770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398712    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.398770 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.398741    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.499851 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.499817    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.499861    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.499905    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dps8s\" (UniqueName: \"kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.499924    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.499960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500006 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.500004    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500314 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.500277    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500381 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.500312    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.500381 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.500372    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.502290 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.502266    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.502469 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.502450    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.510948 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.510920    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dps8s\" (UniqueName: \"kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s\") pod \"noten-model-17d9a5-kserve-85cb4f9f67-dk46s\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") " pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.684407 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.684294    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:26.967386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:26.967288    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 15:59:26.970595 ip-10-0-138-241 kubenswrapper[2560]: W0910 15:59:26.970567    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f06c2c7_c50c_4167_a880_1b02176fcae1.slice/crio-b6fe881cbdb33cbbaa5912bdd5cc4b757543ee8bcbacd98eaed754fca415bf3e WatchSource:0}: Error finding container b6fe881cbdb33cbbaa5912bdd5cc4b757543ee8bcbacd98eaed754fca415bf3e: Status 404 returned error can't find the container with id b6fe881cbdb33cbbaa5912bdd5cc4b757543ee8bcbacd98eaed754fca415bf3e
Sep 10 15:59:27.230659 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:27.230618    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" event={"ID":"0f06c2c7-c50c-4167-a880-1b02176fcae1","Type":"ContainerStarted","Data":"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"}
Sep 10 15:59:27.230861 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:27.230675    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" event={"ID":"0f06c2c7-c50c-4167-a880-1b02176fcae1","Type":"ContainerStarted","Data":"b6fe881cbdb33cbbaa5912bdd5cc4b757543ee8bcbacd98eaed754fca415bf3e"}
Sep 10 15:59:27.251059 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:27.251011    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" podStartSLOduration=1.250996069 podStartE2EDuration="1.250996069s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 15:59:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 15:59:27.249815892 +0000 UTC m=+1045.629762961" watchObservedRunningTime="2026-09-10 15:59:27.250996069 +0000 UTC m=+1045.630943134"
Sep 10 15:59:36.685364 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:36.685291    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:36.685364 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:36.685379    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:36.697836 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:36.697811    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:37.286741 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:37.286707    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:39.657222 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:39.657182    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:59:42.557696 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:42.557654    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:59:43.752041 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:43.752000    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 15:59:50.216672 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.216642    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:50.217092 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.216851    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" podUID="4e40832e-c20b-4310-997c-3f16a552beb6" containerName="payload-processing" containerID="cri-o://35bb262adc95baf4cd43031a4ab98ef2f0e35183e9f125f967ad2bb7765b25d5" gracePeriod=30
Sep 10 15:59:50.229001 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.228970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:50.229286 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.229214    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" podUID="a5490dfe-7475-459c-8c03-417a753b2a19" containerName="payload-pre-processing" containerID="cri-o://68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56" gracePeriod=30
Sep 10 15:59:50.332509 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.332475    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e40832e-c20b-4310-997c-3f16a552beb6" containerID="35bb262adc95baf4cd43031a4ab98ef2f0e35183e9f125f967ad2bb7765b25d5" exitCode=0
Sep 10 15:59:50.332651 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.332545    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" event={"ID":"4e40832e-c20b-4310-997c-3f16a552beb6","Type":"ContainerDied","Data":"35bb262adc95baf4cd43031a4ab98ef2f0e35183e9f125f967ad2bb7765b25d5"}
Sep 10 15:59:50.494920 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.494894    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:50.497165 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.497148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:50.605270 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605238    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l2822\" (UniqueName: \"kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822\") pod \"4e40832e-c20b-4310-997c-3f16a552beb6\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") "
Sep 10 15:59:50.605494 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605325    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-ng8g9\" (UniqueName: \"kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9\") pod \"a5490dfe-7475-459c-8c03-417a753b2a19\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") "
Sep 10 15:59:50.605494 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605438    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume\") pod \"a5490dfe-7475-459c-8c03-417a753b2a19\" (UID: \"a5490dfe-7475-459c-8c03-417a753b2a19\") "
Sep 10 15:59:50.605494 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605475    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume\") pod \"4e40832e-c20b-4310-997c-3f16a552beb6\" (UID: \"4e40832e-c20b-4310-997c-3f16a552beb6\") "
Sep 10 15:59:50.605817 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605792    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a5490dfe-7475-459c-8c03-417a753b2a19" (UID: "a5490dfe-7475-459c-8c03-417a753b2a19"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:50.606078 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.605935    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e40832e-c20b-4310-997c-3f16a552beb6" (UID: "4e40832e-c20b-4310-997c-3f16a552beb6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 15:59:50.607545 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.607523    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822" (OuterVolumeSpecName: "kube-api-access-l2822") pod "4e40832e-c20b-4310-997c-3f16a552beb6" (UID: "4e40832e-c20b-4310-997c-3f16a552beb6"). InnerVolumeSpecName "kube-api-access-l2822". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:50.607617 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.607546    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9" (OuterVolumeSpecName: "kube-api-access-ng8g9") pod "a5490dfe-7475-459c-8c03-417a753b2a19" (UID: "a5490dfe-7475-459c-8c03-417a753b2a19"). InnerVolumeSpecName "kube-api-access-ng8g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:50.706386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.706329    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ng8g9\" (UniqueName: \"kubernetes.io/projected/a5490dfe-7475-459c-8c03-417a753b2a19-kube-api-access-ng8g9\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:50.706386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.706378    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a5490dfe-7475-459c-8c03-417a753b2a19-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:50.706386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.706387    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e40832e-c20b-4310-997c-3f16a552beb6-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:50.706386 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:50.706395    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2822\" (UniqueName: \"kubernetes.io/projected/4e40832e-c20b-4310-997c-3f16a552beb6-kube-api-access-l2822\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:51.338047 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.338007    2560 generic.go:356] "Generic (PLEG): container finished" podID="a5490dfe-7475-459c-8c03-417a753b2a19" containerID="68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56" exitCode=0
Sep 10 15:59:51.338047 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.338052    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"
Sep 10 15:59:51.338567 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.338086    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" event={"ID":"a5490dfe-7475-459c-8c03-417a753b2a19","Type":"ContainerDied","Data":"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"}
Sep 10 15:59:51.338567 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.338152    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8" event={"ID":"a5490dfe-7475-459c-8c03-417a753b2a19","Type":"ContainerDied","Data":"7480cb6d4c178f7128a436af2906395901d4ed37b5f107c6c6642600c35890c3"}
Sep 10 15:59:51.338567 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.338186    2560 scope.go:117] "RemoveContainer" containerID="68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"
Sep 10 15:59:51.339588 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.339570    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"
Sep 10 15:59:51.339650 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.339581    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq" event={"ID":"4e40832e-c20b-4310-997c-3f16a552beb6","Type":"ContainerDied","Data":"4d141056edc352cf142dbbfece9895559e9d14bb6b078c736f7a56ab0adf3b65"}
Sep 10 15:59:51.348256 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.348238    2560 scope.go:117] "RemoveContainer" containerID="68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"
Sep 10 15:59:51.348618 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:59:51.348596    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56\": container with ID starting with 68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56 not found: ID does not exist" containerID="68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"
Sep 10 15:59:51.348707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.348626    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56"} err="failed to get container status \"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56\": rpc error: code = NotFound desc = could not find container \"68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56\": container with ID starting with 68f6287c3fe4dbcfa51bf28a1577f3d640e3a290aecf6acd96aaf62c75d6cd56 not found: ID does not exist"
Sep 10 15:59:51.348707 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.348643    2560 scope.go:117] "RemoveContainer" containerID="35bb262adc95baf4cd43031a4ab98ef2f0e35183e9f125f967ad2bb7765b25d5"
Sep 10 15:59:51.361196 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.361171    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:51.367809 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.367776    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-9a70cb1a-5cfc47fd4b-bxcm8"]
Sep 10 15:59:51.380763 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.380732    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:51.386526 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:51.386505    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-9a70cb1a-5495547d65-5tqdq"]
Sep 10 15:59:52.115686 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:52.115652    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e40832e-c20b-4310-997c-3f16a552beb6" path="/var/lib/kubelet/pods/4e40832e-c20b-4310-997c-3f16a552beb6/volumes"
Sep 10 15:59:52.116013 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:52.116000    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a5490dfe-7475-459c-8c03-417a753b2a19" path="/var/lib/kubelet/pods/a5490dfe-7475-459c-8c03-417a753b2a19/volumes"
Sep 10 15:59:52.959480 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:52.959399    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 15:59:57.577227 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.577155    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" podUID="9f77d1a0-da27-456a-8938-31f9e719df47" containerName="main" containerID="cri-o://c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869" gracePeriod=45
Sep 10 15:59:57.822394 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.822363    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:59:57.975105 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975073    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975125    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975288 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975151    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975355    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975440 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975417    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:57.975541 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975436    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975541 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975433    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home" (OuterVolumeSpecName: "home") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:57.975541 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975474    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x4dtw\" (UniqueName: \"kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw\") pod \"9f77d1a0-da27-456a-8938-31f9e719df47\" (UID: \"9f77d1a0-da27-456a-8938-31f9e719df47\") "
Sep 10 15:59:57.975647 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975632    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache" (OuterVolumeSpecName: "model-cache") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:57.975776 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975760    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:57.975815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975783    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:57.975815 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.975796    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:57.977635 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.977609    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm" (OuterVolumeSpecName: "dshm") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:57.977757 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.977665    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:57.977757 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:57.977712    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw" (OuterVolumeSpecName: "kube-api-access-x4dtw") pod "9f77d1a0-da27-456a-8938-31f9e719df47" (UID: "9f77d1a0-da27-456a-8938-31f9e719df47"). InnerVolumeSpecName "kube-api-access-x4dtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:58.076919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.076881    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9f77d1a0-da27-456a-8938-31f9e719df47-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:58.076919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.076911    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9f77d1a0-da27-456a-8938-31f9e719df47-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:58.076919 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.076922    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x4dtw\" (UniqueName: \"kubernetes.io/projected/9f77d1a0-da27-456a-8938-31f9e719df47-kube-api-access-x4dtw\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:58.372322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.372223    2560 generic.go:356] "Generic (PLEG): container finished" podID="9f77d1a0-da27-456a-8938-31f9e719df47" containerID="c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869" exitCode=0
Sep 10 15:59:58.372322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.372271    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" event={"ID":"9f77d1a0-da27-456a-8938-31f9e719df47","Type":"ContainerDied","Data":"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"}
Sep 10 15:59:58.372322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.372275    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"
Sep 10 15:59:58.372322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.372299    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7" event={"ID":"9f77d1a0-da27-456a-8938-31f9e719df47","Type":"ContainerDied","Data":"170be3fd6c4998965a2b7ef06cc1029d2e09407ccaf0029793c4d6022b7e18a0"}
Sep 10 15:59:58.372322 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.372315    2560 scope.go:117] "RemoveContainer" containerID="c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"
Sep 10 15:59:58.380696 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.380679    2560 scope.go:117] "RemoveContainer" containerID="c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"
Sep 10 15:59:58.381006 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:59:58.380980    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869\": container with ID starting with c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869 not found: ID does not exist" containerID="c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"
Sep 10 15:59:58.381070 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.381017    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869"} err="failed to get container status \"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869\": rpc error: code = NotFound desc = could not find container \"c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869\": container with ID starting with c66590a2e8d42d384db5ade7d3737fad9ff366b51cd22ee6e38784d0f7c7e869 not found: ID does not exist"
Sep 10 15:59:58.390605 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.390582    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:59:58.395579 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.395557    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-607b9a/test-model-607b9a-kserve-545879fd7f-7qzm7"]
Sep 10 15:59:58.771856 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:58.771813    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" podUID="0f06c2c7-c50c-4167-a880-1b02176fcae1" containerName="main" containerID="cri-o://9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182" gracePeriod=45
Sep 10 15:59:59.021109 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.021082    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:59.187370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187242    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187319    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dps8s\" (UniqueName: \"kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187370 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187369    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187649 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187425    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187649 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187447    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187649 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187505    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home\") pod \"0f06c2c7-c50c-4167-a880-1b02176fcae1\" (UID: \"0f06c2c7-c50c-4167-a880-1b02176fcae1\") "
Sep 10 15:59:59.187649 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187570    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:59.187851 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187748    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.187851 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187791    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:59.187851 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.187813    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home" (OuterVolumeSpecName: "home") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:59.189926 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.189892    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm" (OuterVolumeSpecName: "dshm") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 15:59:59.190049 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.189944    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s" (OuterVolumeSpecName: "kube-api-access-dps8s") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "kube-api-access-dps8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 15:59:59.190049 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.189945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f06c2c7-c50c-4167-a880-1b02176fcae1" (UID: "0f06c2c7-c50c-4167-a880-1b02176fcae1"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 15:59:59.289161 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.289120    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dps8s\" (UniqueName: \"kubernetes.io/projected/0f06c2c7-c50c-4167-a880-1b02176fcae1-kube-api-access-dps8s\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.289161 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.289158    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.289161 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.289168    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.289412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.289176    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f06c2c7-c50c-4167-a880-1b02176fcae1-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.289412 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.289183    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f06c2c7-c50c-4167-a880-1b02176fcae1-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 15:59:59.378136 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.378100    2560 generic.go:356] "Generic (PLEG): container finished" podID="0f06c2c7-c50c-4167-a880-1b02176fcae1" containerID="9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182" exitCode=0
Sep 10 15:59:59.378278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.378149    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"
Sep 10 15:59:59.378278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.378186    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" event={"ID":"0f06c2c7-c50c-4167-a880-1b02176fcae1","Type":"ContainerDied","Data":"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"}
Sep 10 15:59:59.378278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.378225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s" event={"ID":"0f06c2c7-c50c-4167-a880-1b02176fcae1","Type":"ContainerDied","Data":"b6fe881cbdb33cbbaa5912bdd5cc4b757543ee8bcbacd98eaed754fca415bf3e"}
Sep 10 15:59:59.378278 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.378244    2560 scope.go:117] "RemoveContainer" containerID="9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"
Sep 10 15:59:59.386801 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.386786    2560 scope.go:117] "RemoveContainer" containerID="9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"
Sep 10 15:59:59.387074 ip-10-0-138-241 kubenswrapper[2560]: E0910 15:59:59.387057    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182\": container with ID starting with 9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182 not found: ID does not exist" containerID="9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"
Sep 10 15:59:59.387120 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.387084    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182"} err="failed to get container status \"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182\": rpc error: code = NotFound desc = could not find container \"9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182\": container with ID starting with 9d536cd40836cfd22f9417f58084366e3d6f9f36d36f3464a669d582bd015182 not found: ID does not exist"
Sep 10 15:59:59.399769 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.399740    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 15:59:59.402128 ip-10-0-138-241 kubenswrapper[2560]: I0910 15:59:59.402108    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-noten-17d9a5/noten-model-17d9a5-kserve-85cb4f9f67-dk46s"]
Sep 10 16:00:00.115510 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.115481    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f06c2c7-c50c-4167-a880-1b02176fcae1" path="/var/lib/kubelet/pods/0f06c2c7-c50c-4167-a880-1b02176fcae1/volumes"
Sep 10 16:00:00.115887 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.115865    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9f77d1a0-da27-456a-8938-31f9e719df47" path="/var/lib/kubelet/pods/9f77d1a0-da27-456a-8938-31f9e719df47/volumes"
Sep 10 16:00:00.167208 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167171    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"]
Sep 10 16:00:00.167504 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167490    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a5490dfe-7475-459c-8c03-417a753b2a19" containerName="payload-pre-processing"
Sep 10 16:00:00.167552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167506    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a5490dfe-7475-459c-8c03-417a753b2a19" containerName="payload-pre-processing"
Sep 10 16:00:00.167552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167523    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9f77d1a0-da27-456a-8938-31f9e719df47" containerName="main"
Sep 10 16:00:00.167552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167532    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="9f77d1a0-da27-456a-8938-31f9e719df47" containerName="main"
Sep 10 16:00:00.167552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167542    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f06c2c7-c50c-4167-a880-1b02176fcae1" containerName="main"
Sep 10 16:00:00.167552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167548    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f06c2c7-c50c-4167-a880-1b02176fcae1" containerName="main"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167556    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e40832e-c20b-4310-997c-3f16a552beb6" containerName="payload-processing"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167563    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e40832e-c20b-4310-997c-3f16a552beb6" containerName="payload-processing"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167627    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="9f77d1a0-da27-456a-8938-31f9e719df47" containerName="main"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167635    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e40832e-c20b-4310-997c-3f16a552beb6" containerName="payload-processing"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167641    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a5490dfe-7475-459c-8c03-417a753b2a19" containerName="payload-pre-processing"
Sep 10 16:00:00.167858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.167647    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f06c2c7-c50c-4167-a880-1b02176fcae1" containerName="main"
Sep 10 16:00:00.175409 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.175375    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"
Sep 10 16:00:00.177982 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.177948    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:00:00.181374 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.181332    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"]
Sep 10 16:00:00.187694 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.187673    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"
Sep 10 16:00:00.461279 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.461103    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"]
Sep 10 16:00:00.464052 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:00:00.464020    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3cd91102_3afc_41d2_8d18_936f684b2346.slice/crio-3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8 WatchSource:0}: Error finding container 3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8: Status 404 returned error can't find the container with id 3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8
Sep 10 16:00:00.466043 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:00.466023    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:00:01.389387 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:01.389331    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw" event={"ID":"3cd91102-3afc-41d2-8d18-936f684b2346","Type":"ContainerStarted","Data":"3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8"}
Sep 10 16:00:03.400430 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:03.400398    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw" event={"ID":"3cd91102-3afc-41d2-8d18-936f684b2346","Type":"ContainerStarted","Data":"1562147fcdf7ca48dc27ec6984aa239f8a0b532cf3f1abd7909745d272fb2f2a"}
Sep 10 16:00:03.418932 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:03.418877    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw" podStartSLOduration=1.13224211 podStartE2EDuration="3.418862008s" totalImagesPullingTime="2.286619898s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:03.417430316 +0000 UTC m=+1081.797377588" watchObservedRunningTime="2026-09-10 16:00:03.418862008 +0000 UTC m=+1081.798809108"
Sep 10 16:00:04.405886 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:04.405853    2560 generic.go:356] "Generic (PLEG): container finished" podID="3cd91102-3afc-41d2-8d18-936f684b2346" containerID="1562147fcdf7ca48dc27ec6984aa239f8a0b532cf3f1abd7909745d272fb2f2a" exitCode=0
Sep 10 16:00:04.406293 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:04.405918    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw" event={"ID":"3cd91102-3afc-41d2-8d18-936f684b2346","Type":"ContainerDied","Data":"1562147fcdf7ca48dc27ec6984aa239f8a0b532cf3f1abd7909745d272fb2f2a"}
Sep 10 16:00:04.775070 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:04.775031    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:00:05.536718 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:05.536692    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"
Sep 10 16:00:06.416062 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:06.415983    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"
Sep 10 16:00:06.416193 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:06.415979    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw" event={"ID":"3cd91102-3afc-41d2-8d18-936f684b2346","Type":"ContainerDied","Data":"3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8"}
Sep 10 16:00:06.416193 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:06.416112    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3bc28c43df2acf00afbe5b3bea897b5e1c9d80f4e9d1c3aae62c6644709413f8"
Sep 10 16:00:12.479949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.479917    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"]
Sep 10 16:00:12.483951 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.483912    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-b-607b9a-29817600-pjslw"]
Sep 10 16:00:12.513805 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.513772    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 16:00:12.514049 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.514023    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" podUID="3b79d16b-46fc-41ea-a226-e3852c226887" containerName="payload-processing" containerID="cri-o://2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573" gracePeriod=30
Sep 10 16:00:12.525575 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.525550    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 16:00:12.525795 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.525760    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" podUID="1e0f9534-a706-48aa-a93e-33348ca7867d" containerName="payload-pre-processing" containerID="cri-o://5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd" gracePeriod=30
Sep 10 16:00:12.785079 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.785058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 16:00:12.787714 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.787698    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 16:00:12.797292 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hf8hn\" (UniqueName: \"kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn\") pod \"1e0f9534-a706-48aa-a93e-33348ca7867d\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") "
Sep 10 16:00:12.797389 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797308    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume\") pod \"3b79d16b-46fc-41ea-a226-e3852c226887\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") "
Sep 10 16:00:12.797389 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797365    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2kvzw\" (UniqueName: \"kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw\") pod \"3b79d16b-46fc-41ea-a226-e3852c226887\" (UID: \"3b79d16b-46fc-41ea-a226-e3852c226887\") "
Sep 10 16:00:12.797477 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797418    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume\") pod \"1e0f9534-a706-48aa-a93e-33348ca7867d\" (UID: \"1e0f9534-a706-48aa-a93e-33348ca7867d\") "
Sep 10 16:00:12.797719 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797691    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "3b79d16b-46fc-41ea-a226-e3852c226887" (UID: "3b79d16b-46fc-41ea-a226-e3852c226887"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:12.797804 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.797751    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "1e0f9534-a706-48aa-a93e-33348ca7867d" (UID: "1e0f9534-a706-48aa-a93e-33348ca7867d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:00:12.799491 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.799468    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn" (OuterVolumeSpecName: "kube-api-access-hf8hn") pod "1e0f9534-a706-48aa-a93e-33348ca7867d" (UID: "1e0f9534-a706-48aa-a93e-33348ca7867d"). InnerVolumeSpecName "kube-api-access-hf8hn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:12.799567 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.799527    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw" (OuterVolumeSpecName: "kube-api-access-2kvzw") pod "3b79d16b-46fc-41ea-a226-e3852c226887" (UID: "3b79d16b-46fc-41ea-a226-e3852c226887"). InnerVolumeSpecName "kube-api-access-2kvzw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:00:12.898531 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.898497    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1e0f9534-a706-48aa-a93e-33348ca7867d-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:00:12.898531 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.898530    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hf8hn\" (UniqueName: \"kubernetes.io/projected/1e0f9534-a706-48aa-a93e-33348ca7867d-kube-api-access-hf8hn\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:00:12.898745 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.898542    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3b79d16b-46fc-41ea-a226-e3852c226887-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:00:12.898745 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:12.898550    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2kvzw\" (UniqueName: \"kubernetes.io/projected/3b79d16b-46fc-41ea-a226-e3852c226887-kube-api-access-2kvzw\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:00:13.446334 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.446299    2560 generic.go:356] "Generic (PLEG): container finished" podID="3b79d16b-46fc-41ea-a226-e3852c226887" containerID="2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573" exitCode=0
Sep 10 16:00:13.446538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.446357    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"
Sep 10 16:00:13.446538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.446397    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" event={"ID":"3b79d16b-46fc-41ea-a226-e3852c226887","Type":"ContainerDied","Data":"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"}
Sep 10 16:00:13.446538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.446455    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h" event={"ID":"3b79d16b-46fc-41ea-a226-e3852c226887","Type":"ContainerDied","Data":"0ec2fef0c6aa933bb468a4ed3e55a95e1a5365efae4fd892d0012385498cd03c"}
Sep 10 16:00:13.446538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.446484    2560 scope.go:117] "RemoveContainer" containerID="2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"
Sep 10 16:00:13.447884 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.447856    2560 generic.go:356] "Generic (PLEG): container finished" podID="1e0f9534-a706-48aa-a93e-33348ca7867d" containerID="5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd" exitCode=0
Sep 10 16:00:13.448004 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.447909    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"
Sep 10 16:00:13.448004 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.447937    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" event={"ID":"1e0f9534-a706-48aa-a93e-33348ca7867d","Type":"ContainerDied","Data":"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"}
Sep 10 16:00:13.448004 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.447968    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q" event={"ID":"1e0f9534-a706-48aa-a93e-33348ca7867d","Type":"ContainerDied","Data":"dcd8f7fbbb61fdbfc07e7f3df3a6ffc34b46d2bc9205d6dc3a9fd6ac08093cc9"}
Sep 10 16:00:13.455837 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.455817    2560 scope.go:117] "RemoveContainer" containerID="2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"
Sep 10 16:00:13.456128 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:00:13.456112    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573\": container with ID starting with 2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573 not found: ID does not exist" containerID="2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"
Sep 10 16:00:13.456181 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.456134    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573"} err="failed to get container status \"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573\": rpc error: code = NotFound desc = could not find container \"2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573\": container with ID starting with 2fd48da64fd8b9ddd11f4918a0cf294688c605474f1f8e71fb9f7a25d54c2573 not found: ID does not exist"
Sep 10 16:00:13.456181 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.456149    2560 scope.go:117] "RemoveContainer" containerID="5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"
Sep 10 16:00:13.464164 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.464149    2560 scope.go:117] "RemoveContainer" containerID="5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"
Sep 10 16:00:13.464435 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:00:13.464419    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd\": container with ID starting with 5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd not found: ID does not exist" containerID="5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"
Sep 10 16:00:13.464480 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.464441    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd"} err="failed to get container status \"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd\": rpc error: code = NotFound desc = could not find container \"5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd\": container with ID starting with 5df97f8a51b9b111a1852e03dbcbe09a3c0a0fd53d8fc498103824123989a1dd not found: ID does not exist"
Sep 10 16:00:13.469954 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.469933    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 16:00:13.474594 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.474573    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-607b9a-6c57f7787c-zwl2h"]
Sep 10 16:00:13.485648 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.485628    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 16:00:13.491122 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:13.491099    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-607b9a-5c768444d6-d9l9q"]
Sep 10 16:00:14.116140 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:14.116105    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1e0f9534-a706-48aa-a93e-33348ca7867d" path="/var/lib/kubelet/pods/1e0f9534-a706-48aa-a93e-33348ca7867d/volumes"
Sep 10 16:00:14.116542 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:14.116523    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3b79d16b-46fc-41ea-a226-e3852c226887" path="/var/lib/kubelet/pods/3b79d16b-46fc-41ea-a226-e3852c226887/volumes"
Sep 10 16:00:14.116826 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:14.116809    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3cd91102-3afc-41d2-8d18-936f684b2346" path="/var/lib/kubelet/pods/3cd91102-3afc-41d2-8d18-936f684b2346/volumes"
Sep 10 16:00:16.555748 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:16.555710    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:00:18.652416 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652381    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652667    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3cd91102-3afc-41d2-8d18-936f684b2346" containerName="cleanup"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652677    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3cd91102-3afc-41d2-8d18-936f684b2346" containerName="cleanup"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652692    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1e0f9534-a706-48aa-a93e-33348ca7867d" containerName="payload-pre-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652697    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="1e0f9534-a706-48aa-a93e-33348ca7867d" containerName="payload-pre-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652705    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3b79d16b-46fc-41ea-a226-e3852c226887" containerName="payload-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652711    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3b79d16b-46fc-41ea-a226-e3852c226887" containerName="payload-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652763    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="1e0f9534-a706-48aa-a93e-33348ca7867d" containerName="payload-pre-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652770    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3b79d16b-46fc-41ea-a226-e3852c226887" containerName="payload-processing"
Sep 10 16:00:18.652801 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.652776    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3cd91102-3afc-41d2-8d18-936f684b2346" containerName="cleanup"
Sep 10 16:00:18.657058 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.657040    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.659803 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.659782    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-247f70\""
Sep 10 16:00:18.659910 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.659826    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-247f70-dockercfg-hr6tq\""
Sep 10 16:00:18.667385 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.667364    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:00:18.740789 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.740754    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wlpvv\" (UniqueName: \"kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.740976 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.740830    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.841662 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.841620    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wlpvv\" (UniqueName: \"kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.841820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.841679    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.842426 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.842405    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.849960 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.849939    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wlpvv\" (UniqueName: \"kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv\") pod \"payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:18.967788 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:18.967699    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:19.242006 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:19.241971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:00:19.245950 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:00:19.245923    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddf5c614d_b993_498c_bc5e_3b88a587e990.slice/crio-2330ef12aaf0f78943b3bba502b5e4df958e7b766826af689e108c19cef0f5e9 WatchSource:0}: Error finding container 2330ef12aaf0f78943b3bba502b5e4df958e7b766826af689e108c19cef0f5e9: Status 404 returned error can't find the container with id 2330ef12aaf0f78943b3bba502b5e4df958e7b766826af689e108c19cef0f5e9
Sep 10 16:00:19.477093 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:19.477057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" event={"ID":"df5c614d-b993-498c-bc5e-3b88a587e990","Type":"ContainerStarted","Data":"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"}
Sep 10 16:00:19.477289 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:19.477114    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" event={"ID":"df5c614d-b993-498c-bc5e-3b88a587e990","Type":"ContainerStarted","Data":"2330ef12aaf0f78943b3bba502b5e4df958e7b766826af689e108c19cef0f5e9"}
Sep 10 16:00:19.477289 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:19.477142    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:19.493863 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:19.493760    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" podStartSLOduration=1.49374459 podStartE2EDuration="1.49374459s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:19.493092094 +0000 UTC m=+1097.873039376" watchObservedRunningTime="2026-09-10 16:00:19.49374459 +0000 UTC m=+1097.873691655"
Sep 10 16:00:26.674250 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.674207    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:00:26.896676 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.896638    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:00:26.900033 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.900009    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:26.902407 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.902380    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-b52270-dockercfg-82nh7\""
Sep 10 16:00:26.902526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.902380    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-b52270\""
Sep 10 16:00:26.911397 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:26.911368    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:00:27.000136 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.000093    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kqsz\" (UniqueName: \"kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.000314 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.000186    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.100676 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.100631    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.100885 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.100699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kqsz\" (UniqueName: \"kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.101407 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.101381    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.109002 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.108978    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kqsz\" (UniqueName: \"kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz\") pod \"payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.210206 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.210169    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:27.482975 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.482823    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:00:27.485684 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:00:27.485652    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954756ba_e36a_452e_a5b5_e889c909ace9.slice/crio-f3abe47bb568511b030fbacab7240e38843a0c8fe0ce03a01ea0e19b8abc53a4 WatchSource:0}: Error finding container f3abe47bb568511b030fbacab7240e38843a0c8fe0ce03a01ea0e19b8abc53a4: Status 404 returned error can't find the container with id f3abe47bb568511b030fbacab7240e38843a0c8fe0ce03a01ea0e19b8abc53a4
Sep 10 16:00:27.510145 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:27.510081    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" event={"ID":"954756ba-e36a-452e-a5b5-e889c909ace9","Type":"ContainerStarted","Data":"f3abe47bb568511b030fbacab7240e38843a0c8fe0ce03a01ea0e19b8abc53a4"}
Sep 10 16:00:28.515415 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:28.515372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" event={"ID":"954756ba-e36a-452e-a5b5-e889c909ace9","Type":"ContainerStarted","Data":"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"}
Sep 10 16:00:28.515804 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:28.515531    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:28.539258 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:28.539203    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" podStartSLOduration=2.539186078 podStartE2EDuration="2.539186078s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:28.53813342 +0000 UTC m=+1106.918080488" watchObservedRunningTime="2026-09-10 16:00:28.539186078 +0000 UTC m=+1106.919133188"
Sep 10 16:00:30.483538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:30.483506    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:00:39.521858 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:39.521824    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:00:39.584800 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:39.584759    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:00:48.117689 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.117656    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:00:48.122245 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.122227    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.124646 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.124623    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-298d9b-dockercfg-6dpnt\""
Sep 10 16:00:48.124798 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.124778    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-298d9b\""
Sep 10 16:00:48.132508 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.132481    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:00:48.166080 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.166048    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k89ml\" (UniqueName: \"kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.166252 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.166099    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.186115 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.186075    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:00:48.188314 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.188291    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.204206 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.204181    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:00:48.267364 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.267298    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.267364 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.267362    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.267608 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.267448    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7fw42\" (UniqueName: \"kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.267608 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.267496    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k89ml\" (UniqueName: \"kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.268006 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.267986    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.281629 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.281603    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-k89ml\" (UniqueName: \"kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml\") pod \"payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.368640 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.368544    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.368640 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.368613    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7fw42\" (UniqueName: \"kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.369308 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.369281    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.377861 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.377837    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7fw42\" (UniqueName: \"kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42\") pod \"payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.432629 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.432590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:48.497559 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.497526    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:48.707109 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.707026    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:00:48.710085 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:00:48.710061    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod81f242c9_fde5_4812_b7e9_f3786d8403ee.slice/crio-1e3c8682f859db63d3478377e81cdd7ea585fb1760dc174896b691fc68103ee4 WatchSource:0}: Error finding container 1e3c8682f859db63d3478377e81cdd7ea585fb1760dc174896b691fc68103ee4: Status 404 returned error can't find the container with id 1e3c8682f859db63d3478377e81cdd7ea585fb1760dc174896b691fc68103ee4
Sep 10 16:00:48.772228 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:48.772204    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:00:48.774600 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:00:48.774570    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf92f9f06_7a8d_430e_821c_8ef41d89d7e1.slice/crio-3a2c6bd318868025250888cb498171d125b4914eff9d3a6a86863918140cc972 WatchSource:0}: Error finding container 3a2c6bd318868025250888cb498171d125b4914eff9d3a6a86863918140cc972: Status 404 returned error can't find the container with id 3a2c6bd318868025250888cb498171d125b4914eff9d3a6a86863918140cc972
Sep 10 16:00:49.607269 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.607225    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" event={"ID":"f92f9f06-7a8d-430e-821c-8ef41d89d7e1","Type":"ContainerStarted","Data":"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"}
Sep 10 16:00:49.607820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.607293    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:00:49.607820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.607319    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" event={"ID":"f92f9f06-7a8d-430e-821c-8ef41d89d7e1","Type":"ContainerStarted","Data":"3a2c6bd318868025250888cb498171d125b4914eff9d3a6a86863918140cc972"}
Sep 10 16:00:49.608834 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.608804    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" event={"ID":"81f242c9-fde5-4812-b7e9-f3786d8403ee","Type":"ContainerStarted","Data":"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"}
Sep 10 16:00:49.608947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.608855    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" event={"ID":"81f242c9-fde5-4812-b7e9-f3786d8403ee","Type":"ContainerStarted","Data":"1e3c8682f859db63d3478377e81cdd7ea585fb1760dc174896b691fc68103ee4"}
Sep 10 16:00:49.608947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.608887    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:00:49.630186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.630067    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" podStartSLOduration=1.630050003 podStartE2EDuration="1.630050003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:49.629451234 +0000 UTC m=+1128.009398301" watchObservedRunningTime="2026-09-10 16:00:49.630050003 +0000 UTC m=+1128.009997069"
Sep 10 16:00:49.669968 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:49.669909    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" podStartSLOduration=1.669889397 podStartE2EDuration="1.669889397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:00:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:00:49.668469484 +0000 UTC m=+1128.048416573" watchObservedRunningTime="2026-09-10 16:00:49.669889397 +0000 UTC m=+1128.049836463"
Sep 10 16:00:54.356267 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:00:54.356226    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:01:00.616136 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:00.616105    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:01:00.616609 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:00.616179    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:01:01.167737 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.167702    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:01:01.169921 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.169905    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.172128 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.172103    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-108dbb\""
Sep 10 16:01:01.172255 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.172164    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-108dbb-dockercfg-m4n96\""
Sep 10 16:01:01.181613 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.181586    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:01:01.273525 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.273483    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x6hmz\" (UniqueName: \"kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.273719 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.273557    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.374089 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.374045    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x6hmz\" (UniqueName: \"kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.374283 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.374117    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.374792 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.374772    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.385651 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.385613    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x6hmz\" (UniqueName: \"kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz\") pod \"payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.480456 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.480418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:01.758078 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:01.758036    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:01:01.760368 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:01.760325    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2f8a8a7e_abfd_44c2_9e3a_2cb387765ca9.slice/crio-4186e7ffc5d0e5393e85c5df1147cc81567b77226857dc02e60174be20a8b43f WatchSource:0}: Error finding container 4186e7ffc5d0e5393e85c5df1147cc81567b77226857dc02e60174be20a8b43f: Status 404 returned error can't find the container with id 4186e7ffc5d0e5393e85c5df1147cc81567b77226857dc02e60174be20a8b43f
Sep 10 16:01:02.672602 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:02.672568    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" event={"ID":"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9","Type":"ContainerStarted","Data":"1b30f9022b0d245cbc8e6c25738a210fa3a03718539985f648b044e96cbbd1f9"}
Sep 10 16:01:02.672796 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:02.672626    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" event={"ID":"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9","Type":"ContainerStarted","Data":"4186e7ffc5d0e5393e85c5df1147cc81567b77226857dc02e60174be20a8b43f"}
Sep 10 16:01:02.672796 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:02.672700    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:02.690530 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:02.690454    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" podStartSLOduration=1.690440033 podStartE2EDuration="1.690440033s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:02.690224538 +0000 UTC m=+1141.070171603" watchObservedRunningTime="2026-09-10 16:01:02.690440033 +0000 UTC m=+1141.070387098"
Sep 10 16:01:09.768697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:09.768657    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:01:11.705557 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.705517    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-0b9280"]
Sep 10 16:01:11.708581 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.708565    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:11.711786 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.711762    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c44xm\""
Sep 10 16:01:11.720359 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.720312    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-0b9280"]
Sep 10 16:01:11.763154 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.763115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hft2z\" (UniqueName: \"kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z\") pod \"test-curl-7517-0b9280\" (UID: \"ed6f7bee-f807-4aa6-90aa-db432cc166da\") " pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:11.864527 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.864485    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hft2z\" (UniqueName: \"kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z\") pod \"test-curl-7517-0b9280\" (UID: \"ed6f7bee-f807-4aa6-90aa-db432cc166da\") " pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:11.879848 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:11.879818    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hft2z\" (UniqueName: \"kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z\") pod \"test-curl-7517-0b9280\" (UID: \"ed6f7bee-f807-4aa6-90aa-db432cc166da\") " pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:12.017453 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:12.017418    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:12.298120 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:12.298041    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-0b9280"]
Sep 10 16:01:12.302133 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:12.302099    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poded6f7bee_f807_4aa6_90aa_db432cc166da.slice/crio-78fec8525a37bda4c984d1745b3649377b6d6d5caac2183eb7b8923ac7ddb463 WatchSource:0}: Error finding container 78fec8525a37bda4c984d1745b3649377b6d6d5caac2183eb7b8923ac7ddb463: Status 404 returned error can't find the container with id 78fec8525a37bda4c984d1745b3649377b6d6d5caac2183eb7b8923ac7ddb463
Sep 10 16:01:12.720690 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:12.720661    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-0b9280" event={"ID":"ed6f7bee-f807-4aa6-90aa-db432cc166da","Type":"ContainerStarted","Data":"253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b"}
Sep 10 16:01:12.721136 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:12.720698    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-0b9280" event={"ID":"ed6f7bee-f807-4aa6-90aa-db432cc166da","Type":"ContainerStarted","Data":"78fec8525a37bda4c984d1745b3649377b6d6d5caac2183eb7b8923ac7ddb463"}
Sep 10 16:01:12.738296 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:12.738243    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7517-0b9280" podStartSLOduration=1.7382267759999999 podStartE2EDuration="1.738226776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:12.737079768 +0000 UTC m=+1151.117026834" watchObservedRunningTime="2026-09-10 16:01:12.738226776 +0000 UTC m=+1151.118173839"
Sep 10 16:01:13.363300 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.363261    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-0b9280"]
Sep 10 16:01:13.372401 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.372371    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-0b9280"]
Sep 10 16:01:13.512714 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.512678    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-a5258c"]
Sep 10 16:01:13.513016 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.513003    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" containerName="test-curl-7517-0b9280"
Sep 10 16:01:13.513062 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.513018    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" containerName="test-curl-7517-0b9280"
Sep 10 16:01:13.513097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.513077    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" containerName="test-curl-7517-0b9280"
Sep 10 16:01:13.514964 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.514949    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:13.539407 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.539367    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-a5258c"]
Sep 10 16:01:13.581288 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.581248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhdrf\" (UniqueName: \"kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf\") pod \"test-curl-7517-a5258c\" (UID: \"c47fbbc0-4c9e-4bd2-9983-72fe387da864\") " pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:13.679698 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.679669    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:01:13.682419 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.682400    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhdrf\" (UniqueName: \"kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf\") pod \"test-curl-7517-a5258c\" (UID: \"c47fbbc0-4c9e-4bd2-9983-72fe387da864\") " pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:13.690932 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.690911    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhdrf\" (UniqueName: \"kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf\") pod \"test-curl-7517-a5258c\" (UID: \"c47fbbc0-4c9e-4bd2-9983-72fe387da864\") " pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:13.725270 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:13.725237    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7517-0b9280" container="test-curl-7517-0b9280"
Sep 10 16:01:13.824082 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:13.823989    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:14.095274 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:14.095250    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-a5258c"]
Sep 10 16:01:14.097675 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:14.097651    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podc47fbbc0_4c9e_4bd2_9983_72fe387da864.slice/crio-b09014a42c14ac1b14a4fba3f08a6ccb50766ba334a64590e313fc95ac193948 WatchSource:0}: Error finding container b09014a42c14ac1b14a4fba3f08a6ccb50766ba334a64590e313fc95ac193948: Status 404 returned error can't find the container with id b09014a42c14ac1b14a4fba3f08a6ccb50766ba334a64590e313fc95ac193948
Sep 10 16:01:14.730419 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:14.730378    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-a5258c" event={"ID":"c47fbbc0-4c9e-4bd2-9983-72fe387da864","Type":"ContainerStarted","Data":"34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436"}
Sep 10 16:01:14.730419 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:14.730423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-a5258c" event={"ID":"c47fbbc0-4c9e-4bd2-9983-72fe387da864","Type":"ContainerStarted","Data":"b09014a42c14ac1b14a4fba3f08a6ccb50766ba334a64590e313fc95ac193948"}
Sep 10 16:01:14.730908 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:14.730418    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7517-0b9280" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" containerName="test-curl-7517-0b9280" containerID="cri-o://253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b" gracePeriod=2
Sep 10 16:01:14.745528 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:14.745473    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7517-a5258c" podStartSLOduration=1.7454573089999998 podStartE2EDuration="1.745457309s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:14.745216724 +0000 UTC m=+1153.125163790" watchObservedRunningTime="2026-09-10 16:01:14.745457309 +0000 UTC m=+1153.125404375"
Sep 10 16:01:15.357325 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.357287    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-a5258c"]
Sep 10 16:01:15.362387 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.362356    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-a5258c"]
Sep 10 16:01:15.477908 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.477872    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-f55f35"]
Sep 10 16:01:15.478218 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.478204    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" containerName="test-curl-7517-a5258c"
Sep 10 16:01:15.478279 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.478223    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" containerName="test-curl-7517-a5258c"
Sep 10 16:01:15.478315 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.478290    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" containerName="test-curl-7517-a5258c"
Sep 10 16:01:15.480408 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.480393    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:15.487432 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.487410    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-f55f35"]
Sep 10 16:01:15.599814 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.599776    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w9hj9\" (UniqueName: \"kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9\") pod \"test-curl-7517-f55f35\" (UID: \"a3e7385d-d3b8-481f-8199-fcd881574478\") " pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:15.700934 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.700891    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w9hj9\" (UniqueName: \"kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9\") pod \"test-curl-7517-f55f35\" (UID: \"a3e7385d-d3b8-481f-8199-fcd881574478\") " pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:15.709970 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.709931    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w9hj9\" (UniqueName: \"kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9\") pod \"test-curl-7517-f55f35\" (UID: \"a3e7385d-d3b8-481f-8199-fcd881574478\") " pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:15.734771 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:15.734692    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7517-a5258c" container="test-curl-7517-a5258c"
Sep 10 16:01:15.790112 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:15.790063    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:16.064192 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.064107    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-f55f35"]
Sep 10 16:01:16.067018 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:16.066988    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda3e7385d_d3b8_481f_8199_fcd881574478.slice/crio-089da6cdba805ccd56edc5aff28d53a59c20d435f9e25b164706d83aa0c4ff29 WatchSource:0}: Error finding container 089da6cdba805ccd56edc5aff28d53a59c20d435f9e25b164706d83aa0c4ff29: Status 404 returned error can't find the container with id 089da6cdba805ccd56edc5aff28d53a59c20d435f9e25b164706d83aa0c4ff29
Sep 10 16:01:16.740938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.740901    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-f55f35" event={"ID":"a3e7385d-d3b8-481f-8199-fcd881574478","Type":"ContainerStarted","Data":"b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f"}
Sep 10 16:01:16.741325 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.740955    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-f55f35" event={"ID":"a3e7385d-d3b8-481f-8199-fcd881574478","Type":"ContainerStarted","Data":"089da6cdba805ccd56edc5aff28d53a59c20d435f9e25b164706d83aa0c4ff29"}
Sep 10 16:01:16.741325 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.741090    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7517-a5258c" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" containerName="test-curl-7517-a5258c" containerID="cri-o://34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436" gracePeriod=2
Sep 10 16:01:16.757718 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.757664    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7517-f55f35" podStartSLOduration=1.7576461060000002 podStartE2EDuration="1.757646106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:16.757274569 +0000 UTC m=+1155.137221637" watchObservedRunningTime="2026-09-10 16:01:16.757646106 +0000 UTC m=+1155.137593173"
Sep 10 16:01:16.874872 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:16.874846    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:17.013306 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.013211    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hft2z\" (UniqueName: \"kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z\") pod \"ed6f7bee-f807-4aa6-90aa-db432cc166da\" (UID: \"ed6f7bee-f807-4aa6-90aa-db432cc166da\") "
Sep 10 16:01:17.015403 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.015377    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z" (OuterVolumeSpecName: "kube-api-access-hft2z") pod "ed6f7bee-f807-4aa6-90aa-db432cc166da" (UID: "ed6f7bee-f807-4aa6-90aa-db432cc166da"). InnerVolumeSpecName "kube-api-access-hft2z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:17.114064 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.114030    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hft2z\" (UniqueName: \"kubernetes.io/projected/ed6f7bee-f807-4aa6-90aa-db432cc166da-kube-api-access-hft2z\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:01:17.351105 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.351017    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-f55f35"]
Sep 10 16:01:17.357484 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.357459    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-f55f35"]
Sep 10 16:01:17.484099 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.484064    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7517-4a9fdb"]
Sep 10 16:01:17.484435 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.484422    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" containerName="test-curl-7517-f55f35"
Sep 10 16:01:17.484491 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.484437    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" containerName="test-curl-7517-f55f35"
Sep 10 16:01:17.484528 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.484500    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" containerName="test-curl-7517-f55f35"
Sep 10 16:01:17.486370 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.486335    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:17.493298 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.493279    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-4a9fdb"]
Sep 10 16:01:17.617483 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.617379    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-225hz\" (UniqueName: \"kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz\") pod \"test-curl-7517-4a9fdb\" (UID: \"e68ba91d-b281-48ba-9d5a-fae17bb0a72a\") " pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:17.718672 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.718630    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-225hz\" (UniqueName: \"kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz\") pod \"test-curl-7517-4a9fdb\" (UID: \"e68ba91d-b281-48ba-9d5a-fae17bb0a72a\") " pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:17.727854 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.727832    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-225hz\" (UniqueName: \"kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz\") pod \"test-curl-7517-4a9fdb\" (UID: \"e68ba91d-b281-48ba-9d5a-fae17bb0a72a\") " pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:17.746821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.746790    2560 generic.go:356] "Generic (PLEG): container finished" podID="ed6f7bee-f807-4aa6-90aa-db432cc166da" containerID="253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b" exitCode=137
Sep 10 16:01:17.747167 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.746838    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:17.747167 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.746895    2560 scope.go:117] "RemoveContainer" containerID="253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b"
Sep 10 16:01:17.747262 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:17.747171    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7517-f55f35" container="test-curl-7517-f55f35"
Sep 10 16:01:17.755475 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.755456    2560 scope.go:117] "RemoveContainer" containerID="253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b"
Sep 10 16:01:17.755745 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:17.755729    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b\": container with ID starting with 253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b not found: ID does not exist" containerID="253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b"
Sep 10 16:01:17.755795 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.755754    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b"} err="failed to get container status \"253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b\": rpc error: code = NotFound desc = could not find container \"253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b\": container with ID starting with 253f6d982f54123cf39bb9eab403f80fbe8c471dd7069ae8303f6e8a0284934b not found: ID does not exist"
Sep 10 16:01:17.757508 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:17.757486    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-0b9280\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" pod="opendatahub/test-curl-7517-0b9280"
Sep 10 16:01:17.795597 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:17.795578    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:18.064139 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.064017    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7517-4a9fdb"]
Sep 10 16:01:18.066644 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:18.066605    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode68ba91d_b281_48ba_9d5a_fae17bb0a72a.slice/crio-74fa02f2117a84bf1ec3fa241ed6d7556c407875b7caeb690e0f531df896f0ca WatchSource:0}: Error finding container 74fa02f2117a84bf1ec3fa241ed6d7556c407875b7caeb690e0f531df896f0ca: Status 404 returned error can't find the container with id 74fa02f2117a84bf1ec3fa241ed6d7556c407875b7caeb690e0f531df896f0ca
Sep 10 16:01:18.115924 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.115894    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ed6f7bee-f807-4aa6-90aa-db432cc166da" path="/var/lib/kubelet/pods/ed6f7bee-f807-4aa6-90aa-db432cc166da/volumes"
Sep 10 16:01:18.754865 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.754835    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-4a9fdb" event={"ID":"e68ba91d-b281-48ba-9d5a-fae17bb0a72a","Type":"ContainerStarted","Data":"f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764"}
Sep 10 16:01:18.755208 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.754880    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7517-4a9fdb" event={"ID":"e68ba91d-b281-48ba-9d5a-fae17bb0a72a","Type":"ContainerStarted","Data":"74fa02f2117a84bf1ec3fa241ed6d7556c407875b7caeb690e0f531df896f0ca"}
Sep 10 16:01:18.755208 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.754875    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7517-f55f35" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" containerName="test-curl-7517-f55f35" containerID="cri-o://b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f" gracePeriod=2
Sep 10 16:01:18.771193 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.771147    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7517-4a9fdb" podStartSLOduration=1.771127685 podStartE2EDuration="1.771127685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:18.769375442 +0000 UTC m=+1157.149322509" watchObservedRunningTime="2026-09-10 16:01:18.771127685 +0000 UTC m=+1157.151074750"
Sep 10 16:01:18.904259 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:18.904230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:19.030636 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.030549    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qhdrf\" (UniqueName: \"kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf\") pod \"c47fbbc0-4c9e-4bd2-9983-72fe387da864\" (UID: \"c47fbbc0-4c9e-4bd2-9983-72fe387da864\") "
Sep 10 16:01:19.032733 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.032702    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf" (OuterVolumeSpecName: "kube-api-access-qhdrf") pod "c47fbbc0-4c9e-4bd2-9983-72fe387da864" (UID: "c47fbbc0-4c9e-4bd2-9983-72fe387da864"). InnerVolumeSpecName "kube-api-access-qhdrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:19.131138 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.131108    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qhdrf\" (UniqueName: \"kubernetes.io/projected/c47fbbc0-4c9e-4bd2-9983-72fe387da864-kube-api-access-qhdrf\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:01:19.355204 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.355117    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7517-4a9fdb"]
Sep 10 16:01:19.362663 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.362626    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7517-4a9fdb"]
Sep 10 16:01:19.760890 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.760857    2560 generic.go:356] "Generic (PLEG): container finished" podID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" containerID="34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436" exitCode=137
Sep 10 16:01:19.761290 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.760917    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:19.761290 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.760973    2560 scope.go:117] "RemoveContainer" containerID="34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436"
Sep 10 16:01:19.761451 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:19.761323    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7517-4a9fdb" container="test-curl-7517-4a9fdb"
Sep 10 16:01:19.769831 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.769817    2560 scope.go:117] "RemoveContainer" containerID="34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436"
Sep 10 16:01:19.770078 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:19.770061    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436\": container with ID starting with 34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436 not found: ID does not exist" containerID="34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436"
Sep 10 16:01:19.770138 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:19.770084    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436"} err="failed to get container status \"34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436\": rpc error: code = NotFound desc = could not find container \"34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436\": container with ID starting with 34660936a5398d7eb0c2727113ee3a37258e639dc9435b74b46afec171586436 not found: ID does not exist"
Sep 10 16:01:19.771159 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:19.771137    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-a5258c\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" pod="opendatahub/test-curl-7517-a5258c"
Sep 10 16:01:20.115855 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:20.115776    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c47fbbc0-4c9e-4bd2-9983-72fe387da864" path="/var/lib/kubelet/pods/c47fbbc0-4c9e-4bd2-9983-72fe387da864/volumes"
Sep 10 16:01:20.766081 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:20.766045    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7517-4a9fdb" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" containerName="test-curl-7517-4a9fdb" containerID="cri-o://f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764" gracePeriod=2
Sep 10 16:01:20.919835 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:20.919804    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:21.049160 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.049070    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w9hj9\" (UniqueName: \"kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9\") pod \"a3e7385d-d3b8-481f-8199-fcd881574478\" (UID: \"a3e7385d-d3b8-481f-8199-fcd881574478\") "
Sep 10 16:01:21.051350 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.051314    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9" (OuterVolumeSpecName: "kube-api-access-w9hj9") pod "a3e7385d-d3b8-481f-8199-fcd881574478" (UID: "a3e7385d-d3b8-481f-8199-fcd881574478"). InnerVolumeSpecName "kube-api-access-w9hj9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:21.149959 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.149922    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w9hj9\" (UniqueName: \"kubernetes.io/projected/a3e7385d-d3b8-481f-8199-fcd881574478-kube-api-access-w9hj9\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:01:21.771292 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.771255    2560 generic.go:356] "Generic (PLEG): container finished" podID="a3e7385d-d3b8-481f-8199-fcd881574478" containerID="b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f" exitCode=137
Sep 10 16:01:21.771739 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.771298    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:21.771739 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.771379    2560 scope.go:117] "RemoveContainer" containerID="b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f"
Sep 10 16:01:21.779800 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.779780    2560 scope.go:117] "RemoveContainer" containerID="b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f"
Sep 10 16:01:21.780091 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:21.780070    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f\": container with ID starting with b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f not found: ID does not exist" containerID="b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f"
Sep 10 16:01:21.780132 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:21.780101    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f"} err="failed to get container status \"b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f\": rpc error: code = NotFound desc = could not find container \"b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f\": container with ID starting with b9809c92783138723a9d177a3c4718550b54ce685a8f33fecfb542d16c9b829f not found: ID does not exist"
Sep 10 16:01:21.781193 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:21.781172    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-f55f35\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:22.114451 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:22.114325    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-f55f35\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" pod="opendatahub/test-curl-7517-f55f35"
Sep 10 16:01:22.115940 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:22.115917    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a3e7385d-d3b8-481f-8199-fcd881574478" path="/var/lib/kubelet/pods/a3e7385d-d3b8-481f-8199-fcd881574478/volumes"
Sep 10 16:01:22.942747 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:22.942720    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:23.066739 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.066633    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-225hz\" (UniqueName: \"kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz\") pod \"e68ba91d-b281-48ba-9d5a-fae17bb0a72a\" (UID: \"e68ba91d-b281-48ba-9d5a-fae17bb0a72a\") "
Sep 10 16:01:23.068903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.068865    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz" (OuterVolumeSpecName: "kube-api-access-225hz") pod "e68ba91d-b281-48ba-9d5a-fae17bb0a72a" (UID: "e68ba91d-b281-48ba-9d5a-fae17bb0a72a"). InnerVolumeSpecName "kube-api-access-225hz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:01:23.167776 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.167662    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-225hz\" (UniqueName: \"kubernetes.io/projected/e68ba91d-b281-48ba-9d5a-fae17bb0a72a-kube-api-access-225hz\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:01:23.783891 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.783853    2560 generic.go:356] "Generic (PLEG): container finished" podID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" containerID="f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764" exitCode=137
Sep 10 16:01:23.784057 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.783898    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:23.784057 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.783940    2560 scope.go:117] "RemoveContainer" containerID="f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764"
Sep 10 16:01:23.792613 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.792592    2560 scope.go:117] "RemoveContainer" containerID="f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764"
Sep 10 16:01:23.792947 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:23.792928    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764\": container with ID starting with f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764 not found: ID does not exist" containerID="f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764"
Sep 10 16:01:23.793005 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:23.792956    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764"} err="failed to get container status \"f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764\": rpc error: code = NotFound desc = could not find container \"f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764\": container with ID starting with f4095a90d723f8c2390e1f256f936643406823924b32e2d96fe14797ec536764 not found: ID does not exist"
Sep 10 16:01:23.793684 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:01:23.793660    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7517-4a9fdb\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" pod="opendatahub/test-curl-7517-4a9fdb"
Sep 10 16:01:24.116120 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:24.116038    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" path="/var/lib/kubelet/pods/e68ba91d-b281-48ba-9d5a-fae17bb0a72a/volumes"
Sep 10 16:01:25.225828 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.225794    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:01:25.226211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.226120    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" containerName="test-curl-7517-4a9fdb"
Sep 10 16:01:25.226211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.226130    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" containerName="test-curl-7517-4a9fdb"
Sep 10 16:01:25.226211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.226207    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e68ba91d-b281-48ba-9d5a-fae17bb0a72a" containerName="test-curl-7517-4a9fdb"
Sep 10 16:01:25.228910 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.228893    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.231246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.231213    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-69972456\""
Sep 10 16:01:25.244142 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.244116    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:01:25.256962 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.256935    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:01:25.259304 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.259281    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.261554 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.261538    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-69972456-dockercfg-fpdxf\""
Sep 10 16:01:25.278949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.278923    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:01:25.386243 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.386210    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xblhv\" (UniqueName: \"kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.386467 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.386249    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5bzq2\" (UniqueName: \"kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.386467 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.386281    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.386467 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.386413    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.487180 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487068    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.487180 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487172    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xblhv\" (UniqueName: \"kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.487481 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487200    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5bzq2\" (UniqueName: \"kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.487481 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487225    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.487859 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487834    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.487938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.487877    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.496761 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.496735    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xblhv\" (UniqueName: \"kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv\") pod \"payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.496950 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.496932    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5bzq2\" (UniqueName: \"kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2\") pod \"payload-processing-e2e-mt-69972456-86d5746bb5-22c2s\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") " pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.538989 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.538948    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:25.568149 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.568116    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:25.807919 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.807893    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:01:25.809699 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:01:25.809674    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10806164_6ab7_4576_b566_664380203eb4.slice/crio-35bb18f806fa3b57d654d82302368cbf8f816c5c37d5e2b975eb840a388f13aa WatchSource:0}: Error finding container 35bb18f806fa3b57d654d82302368cbf8f816c5c37d5e2b975eb840a388f13aa: Status 404 returned error can't find the container with id 35bb18f806fa3b57d654d82302368cbf8f816c5c37d5e2b975eb840a388f13aa
Sep 10 16:01:25.848629 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:25.846001    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:01:26.800570 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.800529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" event={"ID":"0fd96f6e-8576-42a0-bab0-74a4942b87d3","Type":"ContainerStarted","Data":"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"}
Sep 10 16:01:26.801015 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.800593    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" event={"ID":"0fd96f6e-8576-42a0-bab0-74a4942b87d3","Type":"ContainerStarted","Data":"fd1db83691a3e38aa0a891dda12193ade79c2ff7bbfdffbc367287896df6297b"}
Sep 10 16:01:26.801015 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.800622    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:26.802082 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.802056    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" event={"ID":"10806164-6ab7-4576-b566-664380203eb4","Type":"ContainerStarted","Data":"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"}
Sep 10 16:01:26.802191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.802104    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" event={"ID":"10806164-6ab7-4576-b566-664380203eb4","Type":"ContainerStarted","Data":"35bb18f806fa3b57d654d82302368cbf8f816c5c37d5e2b975eb840a388f13aa"}
Sep 10 16:01:26.802191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.802127    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:26.819280 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.819238    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" podStartSLOduration=1.819224716 podStartE2EDuration="1.819224716s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:26.818229008 +0000 UTC m=+1165.198176086" watchObservedRunningTime="2026-09-10 16:01:26.819224716 +0000 UTC m=+1165.199171778"
Sep 10 16:01:26.834686 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:26.834634    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" podStartSLOduration=1.8346196849999998 podStartE2EDuration="1.834619685s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:01:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:01:26.833977483 +0000 UTC m=+1165.213924551" watchObservedRunningTime="2026-09-10 16:01:26.834619685 +0000 UTC m=+1165.214566751"
Sep 10 16:01:29.066349 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:29.066301    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:01:37.809040 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:37.809010    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:01:37.809472 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:37.809084    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:01:52.255401 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:01:52.255361    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:02:06.458306 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.458272    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:02:06.462940 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.462925    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.465192 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.465168    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-69972456\"/\"kube-root-ca.crt\""
Sep 10 16:02:06.466186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.466167    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-69972456\"/\"default-dockercfg-nndt4\""
Sep 10 16:02:06.466288 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.466209    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-69972456\"/\"openshift-service-ca.crt\""
Sep 10 16:02:06.466288 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.466236    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-69972456\"/\"e2e-lifecycle-model-69972456-kserve-self-signed-certs\""
Sep 10 16:02:06.472562 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.472542    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:02:06.544269 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544236    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.544462 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544295    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.544462 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544319    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.544462 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544363    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.544462 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544414    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9s8xs\" (UniqueName: \"kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.544462 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.544442    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.645852 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.645813    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.645852 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.645859    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646073 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.645910    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646073 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.645931    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646073 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.645960    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646073 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.646049    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9s8xs\" (UniqueName: \"kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646351 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.646319    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646485 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.646335    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.646485 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.646438    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.648332 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.648313    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.648623 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.648604    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.655206 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.655177    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9s8xs\" (UniqueName: \"kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs\") pod \"e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") " pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:06.773230 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:06.773191    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:07.049554 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:07.049527    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:02:07.052128 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:02:07.052098    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod41dbac4f_6fba_40bd_b0c2_1d0dba5a91a0.slice/crio-a2e0e286c323b4e8211c1102220358d6d75d12620e638840e81ec71ac28fc25b WatchSource:0}: Error finding container a2e0e286c323b4e8211c1102220358d6d75d12620e638840e81ec71ac28fc25b: Status 404 returned error can't find the container with id a2e0e286c323b4e8211c1102220358d6d75d12620e638840e81ec71ac28fc25b
Sep 10 16:02:07.991571 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:07.991532    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" event={"ID":"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0","Type":"ContainerStarted","Data":"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"}
Sep 10 16:02:07.991571 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:07.991574    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" event={"ID":"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0","Type":"ContainerStarted","Data":"a2e0e286c323b4e8211c1102220358d6d75d12620e638840e81ec71ac28fc25b"}
Sep 10 16:02:08.012014 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:08.011962    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" podStartSLOduration=2.011935258 podStartE2EDuration="2.011935258s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:02:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:02:08.010252109 +0000 UTC m=+1206.390199175" watchObservedRunningTime="2026-09-10 16:02:08.011935258 +0000 UTC m=+1206.391882324"
Sep 10 16:02:11.465188 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:11.465156    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:02:16.774086 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:16.774045    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:16.774517 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:16.774333    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:16.786911 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:16.786882    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:17.043225 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:17.043135    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:02:26.957469 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:26.957429    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:02:50.568334 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:02:50.568295    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:03:02.876523 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:02.876433    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:03:14.376919 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.376883    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:03:14.377527 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.377117    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" podUID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" containerName="payload-processing" containerID="cri-o://ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7" gracePeriod=30
Sep 10 16:03:14.382363 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.382303    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:03:14.382628 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.382571    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" podUID="10806164-6ab7-4576-b566-664380203eb4" containerName="payload-pre-processing" containerID="cri-o://6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6" gracePeriod=30
Sep 10 16:03:14.655186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.655160    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:03:14.657940 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.657911    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:03:14.842501 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842468    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume\") pod \"10806164-6ab7-4576-b566-664380203eb4\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") "
Sep 10 16:03:14.842707 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842519    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5bzq2\" (UniqueName: \"kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2\") pod \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") "
Sep 10 16:03:14.842707 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842570    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xblhv\" (UniqueName: \"kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv\") pod \"10806164-6ab7-4576-b566-664380203eb4\" (UID: \"10806164-6ab7-4576-b566-664380203eb4\") "
Sep 10 16:03:14.842707 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842617    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume\") pod \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\" (UID: \"0fd96f6e-8576-42a0-bab0-74a4942b87d3\") "
Sep 10 16:03:14.842950 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842922    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "10806164-6ab7-4576-b566-664380203eb4" (UID: "10806164-6ab7-4576-b566-664380203eb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:03:14.843019 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.842999    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fd96f6e-8576-42a0-bab0-74a4942b87d3" (UID: "0fd96f6e-8576-42a0-bab0-74a4942b87d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:03:14.844886 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.844858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv" (OuterVolumeSpecName: "kube-api-access-xblhv") pod "10806164-6ab7-4576-b566-664380203eb4" (UID: "10806164-6ab7-4576-b566-664380203eb4"). InnerVolumeSpecName "kube-api-access-xblhv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:14.844989 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.844895    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2" (OuterVolumeSpecName: "kube-api-access-5bzq2") pod "0fd96f6e-8576-42a0-bab0-74a4942b87d3" (UID: "0fd96f6e-8576-42a0-bab0-74a4942b87d3"). InnerVolumeSpecName "kube-api-access-5bzq2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:14.943748 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.943657    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/10806164-6ab7-4576-b566-664380203eb4-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:14.943748 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.943688    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5bzq2\" (UniqueName: \"kubernetes.io/projected/0fd96f6e-8576-42a0-bab0-74a4942b87d3-kube-api-access-5bzq2\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:14.943748 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.943698    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xblhv\" (UniqueName: \"kubernetes.io/projected/10806164-6ab7-4576-b566-664380203eb4-kube-api-access-xblhv\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:14.943748 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:14.943706    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fd96f6e-8576-42a0-bab0-74a4942b87d3-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:15.305045 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.305012    2560 generic.go:356] "Generic (PLEG): container finished" podID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" containerID="ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7" exitCode=0
Sep 10 16:03:15.305220 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.305058    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"
Sep 10 16:03:15.305220 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.305106    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" event={"ID":"0fd96f6e-8576-42a0-bab0-74a4942b87d3","Type":"ContainerDied","Data":"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"}
Sep 10 16:03:15.305220 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.305172    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s" event={"ID":"0fd96f6e-8576-42a0-bab0-74a4942b87d3","Type":"ContainerDied","Data":"fd1db83691a3e38aa0a891dda12193ade79c2ff7bbfdffbc367287896df6297b"}
Sep 10 16:03:15.305220 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.305202    2560 scope.go:117] "RemoveContainer" containerID="ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"
Sep 10 16:03:15.306538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.306515    2560 generic.go:356] "Generic (PLEG): container finished" podID="10806164-6ab7-4576-b566-664380203eb4" containerID="6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6" exitCode=0
Sep 10 16:03:15.306674 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.306552    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" event={"ID":"10806164-6ab7-4576-b566-664380203eb4","Type":"ContainerDied","Data":"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"}
Sep 10 16:03:15.306674 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.306569    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"
Sep 10 16:03:15.306674 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.306603    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g" event={"ID":"10806164-6ab7-4576-b566-664380203eb4","Type":"ContainerDied","Data":"35bb18f806fa3b57d654d82302368cbf8f816c5c37d5e2b975eb840a388f13aa"}
Sep 10 16:03:15.314168 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.314147    2560 scope.go:117] "RemoveContainer" containerID="ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"
Sep 10 16:03:15.314444 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:03:15.314423    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7\": container with ID starting with ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7 not found: ID does not exist" containerID="ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"
Sep 10 16:03:15.314528 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.314451    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7"} err="failed to get container status \"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7\": rpc error: code = NotFound desc = could not find container \"ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7\": container with ID starting with ab1bb4fb0db2ebc135c5f238c5e510b894c4cf4a80b12ffb220474bbcd6c2aa7 not found: ID does not exist"
Sep 10 16:03:15.314528 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.314469    2560 scope.go:117] "RemoveContainer" containerID="6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"
Sep 10 16:03:15.322931 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.322911    2560 scope.go:117] "RemoveContainer" containerID="6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"
Sep 10 16:03:15.323197 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:03:15.323178    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6\": container with ID starting with 6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6 not found: ID does not exist" containerID="6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"
Sep 10 16:03:15.323246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.323202    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6"} err="failed to get container status \"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6\": rpc error: code = NotFound desc = could not find container \"6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6\": container with ID starting with 6f60a4afb352c1c99f0f1b7a88185f5f527ed74b4968ed7052355a327c5354c6 not found: ID does not exist"
Sep 10 16:03:15.331150 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.331130    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:03:15.335334 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.335308    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-69972456-6b56f7c846-46m2g"]
Sep 10 16:03:15.346578 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.346553    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:03:15.350966 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:15.350948    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-69972456-86d5746bb5-22c2s"]
Sep 10 16:03:16.115611 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:16.115582    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" path="/var/lib/kubelet/pods/0fd96f6e-8576-42a0-bab0-74a4942b87d3/volumes"
Sep 10 16:03:16.115984 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:16.115911    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="10806164-6ab7-4576-b566-664380203eb4" path="/var/lib/kubelet/pods/10806164-6ab7-4576-b566-664380203eb4/volumes"
Sep 10 16:03:20.363276 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:20.363236    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:03:22.875059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:22.875027    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:03:35.556694 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:35.556639    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:03:37.895299 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:37.895239    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" podUID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" containerName="main" containerID="cri-o://464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5" gracePeriod=45
Sep 10 16:03:38.139619 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.139589    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:03:38.221799 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.221772    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.221944 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.221840    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.221944 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.221883    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.221944 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.221903    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.222078 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.221953    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9s8xs\" (UniqueName: \"kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.222078 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222000    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm\") pod \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\" (UID: \"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0\") "
Sep 10 16:03:38.222197 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222104    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:38.222257 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222192    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache" (OuterVolumeSpecName: "model-cache") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:38.222257 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222202    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home" (OuterVolumeSpecName: "home") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:38.222394 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222308    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.222394 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222322    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.222394 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.222330    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.224177 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.224156    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm" (OuterVolumeSpecName: "dshm") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:03:38.224278 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.224256    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:03:38.224315 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.224293    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs" (OuterVolumeSpecName: "kube-api-access-9s8xs") pod "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" (UID: "41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0"). InnerVolumeSpecName "kube-api-access-9s8xs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:03:38.322712 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.322679    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.322712 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.322707    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9s8xs\" (UniqueName: \"kubernetes.io/projected/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-kube-api-access-9s8xs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.322712 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.322717    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:03:38.417428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.417395    2560 generic.go:356] "Generic (PLEG): container finished" podID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" containerID="464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5" exitCode=0
Sep 10 16:03:38.417559 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.417439    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"
Sep 10 16:03:38.417559 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.417467    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" event={"ID":"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0","Type":"ContainerDied","Data":"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"}
Sep 10 16:03:38.417559 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.417512    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm" event={"ID":"41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0","Type":"ContainerDied","Data":"a2e0e286c323b4e8211c1102220358d6d75d12620e638840e81ec71ac28fc25b"}
Sep 10 16:03:38.417559 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.417535    2560 scope.go:117] "RemoveContainer" containerID="464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"
Sep 10 16:03:38.426391 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.426373    2560 scope.go:117] "RemoveContainer" containerID="464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"
Sep 10 16:03:38.426678 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:03:38.426659    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5\": container with ID starting with 464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5 not found: ID does not exist" containerID="464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"
Sep 10 16:03:38.426727 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.426686    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5"} err="failed to get container status \"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5\": rpc error: code = NotFound desc = could not find container \"464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5\": container with ID starting with 464c5409d4d9d2f7c9e03d99ac343124c7692d499fdcb10b234863db767a70f5 not found: ID does not exist"
Sep 10 16:03:38.440725 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.440703    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:03:38.444732 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:38.444711    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-69972456/e2e-lifecycle-model-69972456-kserve-6cbc4dd555-mt8qm"]
Sep 10 16:03:40.115723 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:40.115682    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" path="/var/lib/kubelet/pods/41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0/volumes"
Sep 10 16:03:52.358500 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:03:52.358463    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:04:11.965428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:11.965387    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:04:27.797127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:27.797050    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:04:37.870862 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.870829    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-4a62de"]
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871134    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="10806164-6ab7-4576-b566-664380203eb4" containerName="payload-pre-processing"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871145    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="10806164-6ab7-4576-b566-664380203eb4" containerName="payload-pre-processing"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871156    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" containerName="main"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871163    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" containerName="main"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871178    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" containerName="payload-processing"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871183    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" containerName="payload-processing"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871235    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="10806164-6ab7-4576-b566-664380203eb4" containerName="payload-pre-processing"
Sep 10 16:04:37.871240 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871241    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="41dbac4f-6fba-40bd-b0c2-1d0dba5a91a0" containerName="main"
Sep 10 16:04:37.871522 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.871249    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fd96f6e-8576-42a0-bab0-74a4942b87d3" containerName="payload-processing"
Sep 10 16:04:37.874038 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.874023    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:37.876627 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.876608    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c44xm\""
Sep 10 16:04:37.879035 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.879012    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-4a62de"]
Sep 10 16:04:37.909544 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:37.909518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-frxrf\" (UniqueName: \"kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf\") pod \"test-curl-7520-4a62de\" (UID: \"d178f361-f485-4930-bea5-db2e7e88e7d3\") " pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:38.010606 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.010576    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-frxrf\" (UniqueName: \"kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf\") pod \"test-curl-7520-4a62de\" (UID: \"d178f361-f485-4930-bea5-db2e7e88e7d3\") " pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:38.020570 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.020545    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-frxrf\" (UniqueName: \"kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf\") pod \"test-curl-7520-4a62de\" (UID: \"d178f361-f485-4930-bea5-db2e7e88e7d3\") " pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:38.183755 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.183653    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:38.454238 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.454157    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-4a62de"]
Sep 10 16:04:38.457276 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:04:38.457242    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd178f361_f485_4930_bea5_db2e7e88e7d3.slice/crio-e34f22b1f6acd143827e923fa49eade3e89535d9150ac9b033e53cdda7298855 WatchSource:0}: Error finding container e34f22b1f6acd143827e923fa49eade3e89535d9150ac9b033e53cdda7298855: Status 404 returned error can't find the container with id e34f22b1f6acd143827e923fa49eade3e89535d9150ac9b033e53cdda7298855
Sep 10 16:04:38.685211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.685176    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-4a62de" event={"ID":"d178f361-f485-4930-bea5-db2e7e88e7d3","Type":"ContainerStarted","Data":"885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02"}
Sep 10 16:04:38.685211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.685217    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-4a62de" event={"ID":"d178f361-f485-4930-bea5-db2e7e88e7d3","Type":"ContainerStarted","Data":"e34f22b1f6acd143827e923fa49eade3e89535d9150ac9b033e53cdda7298855"}
Sep 10 16:04:38.704883 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:38.704788    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7520-4a62de" podStartSLOduration=1.7047734239999999 podStartE2EDuration="1.704773424s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:38.70302841 +0000 UTC m=+1357.082975470" watchObservedRunningTime="2026-09-10 16:04:38.704773424 +0000 UTC m=+1357.084720490"
Sep 10 16:04:39.319372 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:39.319312    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-4a62de"]
Sep 10 16:04:39.324628 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:39.324600    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-4a62de"]
Sep 10 16:04:39.689169 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:04:39.689140    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7520-4a62de" container="test-curl-7520-4a62de"
Sep 10 16:04:40.693580 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:40.693543    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7520-4a62de" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" containerName="test-curl-7520-4a62de" containerID="cri-o://885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02" gracePeriod=2
Sep 10 16:04:42.876549 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:42.876518    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:42.953140 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:42.953062    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-frxrf\" (UniqueName: \"kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf\") pod \"d178f361-f485-4930-bea5-db2e7e88e7d3\" (UID: \"d178f361-f485-4930-bea5-db2e7e88e7d3\") "
Sep 10 16:04:42.955294 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:42.955267    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf" (OuterVolumeSpecName: "kube-api-access-frxrf") pod "d178f361-f485-4930-bea5-db2e7e88e7d3" (UID: "d178f361-f485-4930-bea5-db2e7e88e7d3"). InnerVolumeSpecName "kube-api-access-frxrf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:04:43.053679 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.053642    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-frxrf\" (UniqueName: \"kubernetes.io/projected/d178f361-f485-4930-bea5-db2e7e88e7d3-kube-api-access-frxrf\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:04:43.565892 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.565855    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:04:43.716228 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.716185    2560 generic.go:356] "Generic (PLEG): container finished" podID="d178f361-f485-4930-bea5-db2e7e88e7d3" containerID="885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02" exitCode=137
Sep 10 16:04:43.716444 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.716243    2560 scope.go:117] "RemoveContainer" containerID="885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02"
Sep 10 16:04:43.716444 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.716240    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:43.725496 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.725474    2560 scope.go:117] "RemoveContainer" containerID="885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02"
Sep 10 16:04:43.725801 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:04:43.725782    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02\": container with ID starting with 885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02 not found: ID does not exist" containerID="885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02"
Sep 10 16:04:43.725847 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:43.725809    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02"} err="failed to get container status \"885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02\": rpc error: code = NotFound desc = could not find container \"885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02\": container with ID starting with 885497eaeacabfb8a5ceb399118f9e0c064ae60414b33af83032c1d4bdab8a02 not found: ID does not exist"
Sep 10 16:04:43.726957 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:04:43.726933    2560 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7520-4a62de\" is forbidden: User \"system:node:ip-10-0-138-241.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-138-241.ec2.internal' and this object" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" pod="opendatahub/test-curl-7520-4a62de"
Sep 10 16:04:44.116452 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:44.116420    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" path="/var/lib/kubelet/pods/d178f361-f485-4930-bea5-db2e7e88e7d3/volumes"
Sep 10 16:04:58.662708 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.662663    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:04:58.663186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.662994    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" containerName="test-curl-7520-4a62de"
Sep 10 16:04:58.663186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.663006    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" containerName="test-curl-7520-4a62de"
Sep 10 16:04:58.663186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.663057    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="d178f361-f485-4930-bea5-db2e7e88e7d3" containerName="test-curl-7520-4a62de"
Sep 10 16:04:58.667331 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.667313    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.670127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.670102    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-ea0f68-dockercfg-b4m76\""
Sep 10 16:04:58.670408 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.670389    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-ea0f68\""
Sep 10 16:04:58.683705 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.683682    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:04:58.784457 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.784423    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bjp95\" (UniqueName: \"kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.784615 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.784528    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.885740 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.885702    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bjp95\" (UniqueName: \"kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.885912 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.885785    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.886440 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.886421    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.895319 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.895297    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bjp95\" (UniqueName: \"kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95\") pod \"payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") " pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:58.977917 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:58.977880    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:59.256428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:59.256397    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:04:59.259160 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:04:59.259134    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda242cd2a_2fa7_4b87_9af5_3226632f90ca.slice/crio-6a14656bc541a2a63e694ff7623e29e29af5e0621a7674826a3215e70cbdb1e8 WatchSource:0}: Error finding container 6a14656bc541a2a63e694ff7623e29e29af5e0621a7674826a3215e70cbdb1e8: Status 404 returned error can't find the container with id 6a14656bc541a2a63e694ff7623e29e29af5e0621a7674826a3215e70cbdb1e8
Sep 10 16:04:59.786896 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:59.786856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" event={"ID":"a242cd2a-2fa7-4b87-9af5-3226632f90ca","Type":"ContainerStarted","Data":"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"}
Sep 10 16:04:59.787304 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:59.786919    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" event={"ID":"a242cd2a-2fa7-4b87-9af5-3226632f90ca","Type":"ContainerStarted","Data":"6a14656bc541a2a63e694ff7623e29e29af5e0621a7674826a3215e70cbdb1e8"}
Sep 10 16:04:59.787304 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:59.786961    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:04:59.808385 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:04:59.808314    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" podStartSLOduration=1.808297351 podStartE2EDuration="1.808297351s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:04:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:04:59.806507122 +0000 UTC m=+1378.186454188" watchObservedRunningTime="2026-09-10 16:04:59.808297351 +0000 UTC m=+1378.188244416"
Sep 10 16:05:04.670612 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:04.670575    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:05:10.793036 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:10.792999    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:05:19.969468 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:19.969429    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:05:27.937335 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.937302    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:05:27.941714 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.941698    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:27.945230 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.945209    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-2ab381-dockercfg-jh5kf\""
Sep 10 16:05:27.945230 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.945216    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-2ab381\""
Sep 10 16:05:27.958200 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.958174    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:05:27.972311 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.972279    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:05:27.975631 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.975611    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:27.989034 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:27.989010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:05:28.021747 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.021719    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwrrc\" (UniqueName: \"kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.021899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.021765    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xlzdh\" (UniqueName: \"kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.021899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.021786    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.021899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.021835    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.122674 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.122640    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.122857 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.122722    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwrrc\" (UniqueName: \"kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.122857 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.122778    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xlzdh\" (UniqueName: \"kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.122857 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.122810    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.123402 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.123375    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.123542 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.123521    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.134898 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.134866    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xlzdh\" (UniqueName: \"kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh\") pod \"payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") " pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.137505 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.137486    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwrrc\" (UniqueName: \"kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc\") pod \"payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.257265 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.257230    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.285431 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.285398    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.535860 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.535834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:05:28.537558 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:05:28.537532    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00535a56_c76d_456e_aa76_b44d6c69ecce.slice/crio-c19d7007ade6e6b22d8ecd34ea244a5062962179480399340f764ceefc2d3b02 WatchSource:0}: Error finding container c19d7007ade6e6b22d8ecd34ea244a5062962179480399340f764ceefc2d3b02: Status 404 returned error can't find the container with id c19d7007ade6e6b22d8ecd34ea244a5062962179480399340f764ceefc2d3b02
Sep 10 16:05:28.578851 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.578828    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:05:28.581242 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:05:28.581217    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod457061c8_cdd4_46f5_aab9_08409791250d.slice/crio-95a4ce508c99669a392b801faca357afd9edb64b977be4726e6b7511712ab28c WatchSource:0}: Error finding container 95a4ce508c99669a392b801faca357afd9edb64b977be4726e6b7511712ab28c: Status 404 returned error can't find the container with id 95a4ce508c99669a392b801faca357afd9edb64b977be4726e6b7511712ab28c
Sep 10 16:05:28.912163 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.912080    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" event={"ID":"457061c8-cdd4-46f5-aab9-08409791250d","Type":"ContainerStarted","Data":"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"}
Sep 10 16:05:28.912163 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.912136    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" event={"ID":"457061c8-cdd4-46f5-aab9-08409791250d","Type":"ContainerStarted","Data":"95a4ce508c99669a392b801faca357afd9edb64b977be4726e6b7511712ab28c"}
Sep 10 16:05:28.912428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.912173    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:28.913856 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.913829    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" event={"ID":"00535a56-c76d-456e-aa76-b44d6c69ecce","Type":"ContainerStarted","Data":"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"}
Sep 10 16:05:28.913989 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.913873    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" event={"ID":"00535a56-c76d-456e-aa76-b44d6c69ecce","Type":"ContainerStarted","Data":"c19d7007ade6e6b22d8ecd34ea244a5062962179480399340f764ceefc2d3b02"}
Sep 10 16:05:28.913989 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.913935    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:28.935270 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.935216    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" podStartSLOduration=1.9352027710000002 podStartE2EDuration="1.935202771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:28.933964633 +0000 UTC m=+1407.313911704" watchObservedRunningTime="2026-09-10 16:05:28.935202771 +0000 UTC m=+1407.315149836"
Sep 10 16:05:28.961156 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:28.961113    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" podStartSLOduration=1.961096583 podStartE2EDuration="1.961096583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:05:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:05:28.960481558 +0000 UTC m=+1407.340428626" watchObservedRunningTime="2026-09-10 16:05:28.961096583 +0000 UTC m=+1407.341043687"
Sep 10 16:05:36.768160 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:36.768119    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:05:39.920540 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:39.920513    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:05:39.920941 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:39.920583    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:05:55.603210 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:05:55.603112    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:06:02.770965 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:02.770929    2560 scope.go:117] "RemoveContainer" containerID="1562147fcdf7ca48dc27ec6984aa239f8a0b532cf3f1abd7909745d272fb2f2a"
Sep 10 16:06:10.186172 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.186137    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:06:10.186724 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.186461    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" podUID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" containerName="payload-processing" containerID="cri-o://62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266" gracePeriod=30
Sep 10 16:06:10.434895 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.434870    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:06:10.481679 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.481635    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume\") pod \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") "
Sep 10 16:06:10.481862 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.481743    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bjp95\" (UniqueName: \"kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95\") pod \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\" (UID: \"a242cd2a-2fa7-4b87-9af5-3226632f90ca\") "
Sep 10 16:06:10.482008 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.481979    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a242cd2a-2fa7-4b87-9af5-3226632f90ca" (UID: "a242cd2a-2fa7-4b87-9af5-3226632f90ca"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:10.483880 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.483858    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95" (OuterVolumeSpecName: "kube-api-access-bjp95") pod "a242cd2a-2fa7-4b87-9af5-3226632f90ca" (UID: "a242cd2a-2fa7-4b87-9af5-3226632f90ca"). InnerVolumeSpecName "kube-api-access-bjp95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:10.582785 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.582745    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bjp95\" (UniqueName: \"kubernetes.io/projected/a242cd2a-2fa7-4b87-9af5-3226632f90ca-kube-api-access-bjp95\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:10.582785 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:10.582777    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a242cd2a-2fa7-4b87-9af5-3226632f90ca-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:11.108880 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.108841    2560 generic.go:356] "Generic (PLEG): container finished" podID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" containerID="62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266" exitCode=0
Sep 10 16:06:11.109059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.108887    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"
Sep 10 16:06:11.109059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.108902    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" event={"ID":"a242cd2a-2fa7-4b87-9af5-3226632f90ca","Type":"ContainerDied","Data":"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"}
Sep 10 16:06:11.109059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.108954    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f" event={"ID":"a242cd2a-2fa7-4b87-9af5-3226632f90ca","Type":"ContainerDied","Data":"6a14656bc541a2a63e694ff7623e29e29af5e0621a7674826a3215e70cbdb1e8"}
Sep 10 16:06:11.109059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.108984    2560 scope.go:117] "RemoveContainer" containerID="62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"
Sep 10 16:06:11.124103 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.124081    2560 scope.go:117] "RemoveContainer" containerID="62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"
Sep 10 16:06:11.124410 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:11.124384    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266\": container with ID starting with 62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266 not found: ID does not exist" containerID="62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"
Sep 10 16:06:11.124530 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.124422    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266"} err="failed to get container status \"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266\": rpc error: code = NotFound desc = could not find container \"62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266\": container with ID starting with 62dcf00cf9378ba6a4266f8f28688e5811237b5790b732bc517874e04de55266 not found: ID does not exist"
Sep 10 16:06:11.136381 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.136358    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:06:11.142550 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:11.142526    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-ea0f68-5f676f986c-klk5f"]
Sep 10 16:06:12.116456 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:12.116422    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" path="/var/lib/kubelet/pods/a242cd2a-2fa7-4b87-9af5-3226632f90ca/volumes"
Sep 10 16:06:14.398229 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:14.398191    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:06:34.761308 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:34.761269    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:06:35.330807 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.330770    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:06:35.331092 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.331067    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" podUID="457061c8-cdd4-46f5-aab9-08409791250d" containerName="payload-processing" containerID="cri-o://38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c" gracePeriod=30
Sep 10 16:06:35.339130 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.339099    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:06:35.339437 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.339382    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" podUID="00535a56-c76d-456e-aa76-b44d6c69ecce" containerName="payload-pre-processing" containerID="cri-o://f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da" gracePeriod=30
Sep 10 16:06:35.602306 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.602281    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:06:35.605355 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.605315    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:06:35.692712 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.692667    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwrrc\" (UniqueName: \"kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc\") pod \"00535a56-c76d-456e-aa76-b44d6c69ecce\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") "
Sep 10 16:06:35.692907 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.692723    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xlzdh\" (UniqueName: \"kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh\") pod \"457061c8-cdd4-46f5-aab9-08409791250d\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") "
Sep 10 16:06:35.692907 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.692745    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume\") pod \"00535a56-c76d-456e-aa76-b44d6c69ecce\" (UID: \"00535a56-c76d-456e-aa76-b44d6c69ecce\") "
Sep 10 16:06:35.692907 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.692773    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume\") pod \"457061c8-cdd4-46f5-aab9-08409791250d\" (UID: \"457061c8-cdd4-46f5-aab9-08409791250d\") "
Sep 10 16:06:35.693184 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.693155    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "00535a56-c76d-456e-aa76-b44d6c69ecce" (UID: "00535a56-c76d-456e-aa76-b44d6c69ecce"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:35.693259 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.693199    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "457061c8-cdd4-46f5-aab9-08409791250d" (UID: "457061c8-cdd4-46f5-aab9-08409791250d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:35.695061 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.695039    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc" (OuterVolumeSpecName: "kube-api-access-qwrrc") pod "00535a56-c76d-456e-aa76-b44d6c69ecce" (UID: "00535a56-c76d-456e-aa76-b44d6c69ecce"). InnerVolumeSpecName "kube-api-access-qwrrc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:35.695115 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.695042    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh" (OuterVolumeSpecName: "kube-api-access-xlzdh") pod "457061c8-cdd4-46f5-aab9-08409791250d" (UID: "457061c8-cdd4-46f5-aab9-08409791250d"). InnerVolumeSpecName "kube-api-access-xlzdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:35.793948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.793906    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwrrc\" (UniqueName: \"kubernetes.io/projected/00535a56-c76d-456e-aa76-b44d6c69ecce-kube-api-access-qwrrc\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:35.793948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.793941    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xlzdh\" (UniqueName: \"kubernetes.io/projected/457061c8-cdd4-46f5-aab9-08409791250d-kube-api-access-xlzdh\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:35.793948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.793950    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/00535a56-c76d-456e-aa76-b44d6c69ecce-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:35.793948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:35.793960    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/457061c8-cdd4-46f5-aab9-08409791250d-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:36.216056 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.216021    2560 generic.go:356] "Generic (PLEG): container finished" podID="457061c8-cdd4-46f5-aab9-08409791250d" containerID="38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c" exitCode=0
Sep 10 16:06:36.216246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.216066    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"
Sep 10 16:06:36.216246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.216107    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" event={"ID":"457061c8-cdd4-46f5-aab9-08409791250d","Type":"ContainerDied","Data":"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"}
Sep 10 16:06:36.216246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.216165    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5" event={"ID":"457061c8-cdd4-46f5-aab9-08409791250d","Type":"ContainerDied","Data":"95a4ce508c99669a392b801faca357afd9edb64b977be4726e6b7511712ab28c"}
Sep 10 16:06:36.216246 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.216195    2560 scope.go:117] "RemoveContainer" containerID="38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"
Sep 10 16:06:36.217572 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.217545    2560 generic.go:356] "Generic (PLEG): container finished" podID="00535a56-c76d-456e-aa76-b44d6c69ecce" containerID="f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da" exitCode=0
Sep 10 16:06:36.217700 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.217587    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"
Sep 10 16:06:36.217700 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.217597    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" event={"ID":"00535a56-c76d-456e-aa76-b44d6c69ecce","Type":"ContainerDied","Data":"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"}
Sep 10 16:06:36.217700 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.217638    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz" event={"ID":"00535a56-c76d-456e-aa76-b44d6c69ecce","Type":"ContainerDied","Data":"c19d7007ade6e6b22d8ecd34ea244a5062962179480399340f764ceefc2d3b02"}
Sep 10 16:06:36.225206 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.225186    2560 scope.go:117] "RemoveContainer" containerID="38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"
Sep 10 16:06:36.225548 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:36.225523    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c\": container with ID starting with 38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c not found: ID does not exist" containerID="38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"
Sep 10 16:06:36.225649 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.225555    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c"} err="failed to get container status \"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c\": rpc error: code = NotFound desc = could not find container \"38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c\": container with ID starting with 38868ee38012275e98a66aee9b7040ee8f7c7305ea838f4db787e58d9d94f02c not found: ID does not exist"
Sep 10 16:06:36.225649 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.225573    2560 scope.go:117] "RemoveContainer" containerID="f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"
Sep 10 16:06:36.233541 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.233520    2560 scope.go:117] "RemoveContainer" containerID="f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"
Sep 10 16:06:36.233848 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:36.233828    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da\": container with ID starting with f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da not found: ID does not exist" containerID="f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"
Sep 10 16:06:36.233911 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.233860    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da"} err="failed to get container status \"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da\": rpc error: code = NotFound desc = could not find container \"f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da\": container with ID starting with f5385fc9ccd65b7354646d833e3550a7fd88d61383425c769ae15867d97f25da not found: ID does not exist"
Sep 10 16:06:36.237354 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.237315    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:06:36.243521 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.243498    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-2ab381-84495d6b5f-nq4p5"]
Sep 10 16:06:36.254706 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.254680    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:06:36.264352 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:36.264309    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-2ab381-58c75d67d-5rzdz"]
Sep 10 16:06:38.115951 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.115916    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="00535a56-c76d-456e-aa76-b44d6c69ecce" path="/var/lib/kubelet/pods/00535a56-c76d-456e-aa76-b44d6c69ecce/volumes"
Sep 10 16:06:38.116315 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.116239    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="457061c8-cdd4-46f5-aab9-08409791250d" path="/var/lib/kubelet/pods/457061c8-cdd4-46f5-aab9-08409791250d/volumes"
Sep 10 16:06:38.263366 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263303    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-f111e6"]
Sep 10 16:06:38.263635 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263621    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="457061c8-cdd4-46f5-aab9-08409791250d" containerName="payload-processing"
Sep 10 16:06:38.263678 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263637    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="457061c8-cdd4-46f5-aab9-08409791250d" containerName="payload-processing"
Sep 10 16:06:38.263678 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263651    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" containerName="payload-processing"
Sep 10 16:06:38.263678 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263657    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" containerName="payload-processing"
Sep 10 16:06:38.263678 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263669    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="00535a56-c76d-456e-aa76-b44d6c69ecce" containerName="payload-pre-processing"
Sep 10 16:06:38.263678 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263674    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="00535a56-c76d-456e-aa76-b44d6c69ecce" containerName="payload-pre-processing"
Sep 10 16:06:38.263830 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263725    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="00535a56-c76d-456e-aa76-b44d6c69ecce" containerName="payload-pre-processing"
Sep 10 16:06:38.263830 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263733    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a242cd2a-2fa7-4b87-9af5-3226632f90ca" containerName="payload-processing"
Sep 10 16:06:38.263830 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.263740    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="457061c8-cdd4-46f5-aab9-08409791250d" containerName="payload-processing"
Sep 10 16:06:38.267991 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.267966    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:38.270558 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.270540    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-c44xm\""
Sep 10 16:06:38.276629 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.276606    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-f111e6"]
Sep 10 16:06:38.416821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.416724    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgf4w\" (UniqueName: \"kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w\") pod \"test-curl-7520-f111e6\" (UID: \"dca7dddb-5921-42d5-ae20-004bcbc3439a\") " pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:38.517439 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.517395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgf4w\" (UniqueName: \"kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w\") pod \"test-curl-7520-f111e6\" (UID: \"dca7dddb-5921-42d5-ae20-004bcbc3439a\") " pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:38.533731 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.533690    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgf4w\" (UniqueName: \"kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w\") pod \"test-curl-7520-f111e6\" (UID: \"dca7dddb-5921-42d5-ae20-004bcbc3439a\") " pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:38.578179 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.578137    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:38.848376 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:38.848326    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-f111e6"]
Sep 10 16:06:38.854813 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:06:38.854784    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddca7dddb_5921_42d5_ae20_004bcbc3439a.slice/crio-e70d7c6e1710198d8ea8ec725988ec3b3b72bbb3d8571841d2ae9e1f99b0cc8c WatchSource:0}: Error finding container e70d7c6e1710198d8ea8ec725988ec3b3b72bbb3d8571841d2ae9e1f99b0cc8c: Status 404 returned error can't find the container with id e70d7c6e1710198d8ea8ec725988ec3b3b72bbb3d8571841d2ae9e1f99b0cc8c
Sep 10 16:06:39.234207 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.234166    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-f111e6" event={"ID":"dca7dddb-5921-42d5-ae20-004bcbc3439a","Type":"ContainerStarted","Data":"8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445"}
Sep 10 16:06:39.234207 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.234209    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-f111e6" event={"ID":"dca7dddb-5921-42d5-ae20-004bcbc3439a","Type":"ContainerStarted","Data":"e70d7c6e1710198d8ea8ec725988ec3b3b72bbb3d8571841d2ae9e1f99b0cc8c"}
Sep 10 16:06:39.251016 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.250961    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7520-f111e6" podStartSLOduration=1.250943251 podStartE2EDuration="1.250943251s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:39.249970268 +0000 UTC m=+1477.629917348" watchObservedRunningTime="2026-09-10 16:06:39.250943251 +0000 UTC m=+1477.630890317"
Sep 10 16:06:39.845930 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.845896    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-f111e6"]
Sep 10 16:06:39.849854 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.849822    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-f111e6"]
Sep 10 16:06:39.965683 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.965598    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7520-a13bea"]
Sep 10 16:06:39.965949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.965931    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dca7dddb-5921-42d5-ae20-004bcbc3439a" containerName="test-curl-7520-f111e6"
Sep 10 16:06:39.965992 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.965953    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="dca7dddb-5921-42d5-ae20-004bcbc3439a" containerName="test-curl-7520-f111e6"
Sep 10 16:06:39.966027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.966012    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="dca7dddb-5921-42d5-ae20-004bcbc3439a" containerName="test-curl-7520-f111e6"
Sep 10 16:06:39.968931 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.968914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:39.976158 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:39.976132    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-a13bea"]
Sep 10 16:06:40.029682 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:40.029637    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-458dd\" (UniqueName: \"kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd\") pod \"test-curl-7520-a13bea\" (UID: \"e1040edb-5ac8-4be5-8f98-d2b3c9764302\") " pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:40.130445 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:40.130404    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-458dd\" (UniqueName: \"kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd\") pod \"test-curl-7520-a13bea\" (UID: \"e1040edb-5ac8-4be5-8f98-d2b3c9764302\") " pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:40.139476 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:40.139449    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-458dd\" (UniqueName: \"kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd\") pod \"test-curl-7520-a13bea\" (UID: \"e1040edb-5ac8-4be5-8f98-d2b3c9764302\") " pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:40.239157 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:40.239116    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7520-f111e6" container="test-curl-7520-f111e6"
Sep 10 16:06:40.278818 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:40.278765    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:40.550519 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:40.550493    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7520-a13bea"]
Sep 10 16:06:40.552760 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:06:40.552735    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode1040edb_5ac8_4be5_8f98_d2b3c9764302.slice/crio-3b4e115d3b3646aeb1152ba15874979f1d73780a17d196c58aa953574b7b2bfc WatchSource:0}: Error finding container 3b4e115d3b3646aeb1152ba15874979f1d73780a17d196c58aa953574b7b2bfc: Status 404 returned error can't find the container with id 3b4e115d3b3646aeb1152ba15874979f1d73780a17d196c58aa953574b7b2bfc
Sep 10 16:06:41.245261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.245218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-a13bea" event={"ID":"e1040edb-5ac8-4be5-8f98-d2b3c9764302","Type":"ContainerStarted","Data":"4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5"}
Sep 10 16:06:41.245261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.245263    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7520-a13bea" event={"ID":"e1040edb-5ac8-4be5-8f98-d2b3c9764302","Type":"ContainerStarted","Data":"3b4e115d3b3646aeb1152ba15874979f1d73780a17d196c58aa953574b7b2bfc"}
Sep 10 16:06:41.245857 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.245298    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7520-f111e6" podUID="dca7dddb-5921-42d5-ae20-004bcbc3439a" containerName="test-curl-7520-f111e6" containerID="cri-o://8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445" gracePeriod=2
Sep 10 16:06:41.266093 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.266032    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7520-a13bea" podStartSLOduration=2.266013152 podStartE2EDuration="2.266013152s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:06:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:06:41.264375654 +0000 UTC m=+1479.644322721" watchObservedRunningTime="2026-09-10 16:06:41.266013152 +0000 UTC m=+1479.645960218"
Sep 10 16:06:41.871449 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.871365    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7520-a13bea"]
Sep 10 16:06:41.873583 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:41.873543    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7520-a13bea"]
Sep 10 16:06:42.249975 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:42.249943    2560 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7520-a13bea" container="test-curl-7520-a13bea"
Sep 10 16:06:43.256627 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:43.256593    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7520-a13bea" podUID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" containerName="test-curl-7520-a13bea" containerID="cri-o://4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5" gracePeriod=2
Sep 10 16:06:43.428293 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:43.428268    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:43.565599 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:43.565509    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgf4w\" (UniqueName: \"kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w\") pod \"dca7dddb-5921-42d5-ae20-004bcbc3439a\" (UID: \"dca7dddb-5921-42d5-ae20-004bcbc3439a\") "
Sep 10 16:06:43.567769 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:43.567736    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w" (OuterVolumeSpecName: "kube-api-access-tgf4w") pod "dca7dddb-5921-42d5-ae20-004bcbc3439a" (UID: "dca7dddb-5921-42d5-ae20-004bcbc3439a"). InnerVolumeSpecName "kube-api-access-tgf4w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:43.666420 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:43.666382    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgf4w\" (UniqueName: \"kubernetes.io/projected/dca7dddb-5921-42d5-ae20-004bcbc3439a-kube-api-access-tgf4w\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:44.116233 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.116194    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dca7dddb-5921-42d5-ae20-004bcbc3439a" path="/var/lib/kubelet/pods/dca7dddb-5921-42d5-ae20-004bcbc3439a/volumes"
Sep 10 16:06:44.262495 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.262456    2560 generic.go:356] "Generic (PLEG): container finished" podID="dca7dddb-5921-42d5-ae20-004bcbc3439a" containerID="8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445" exitCode=137
Sep 10 16:06:44.262495 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.262499    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-f111e6"
Sep 10 16:06:44.263003 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.262556    2560 scope.go:117] "RemoveContainer" containerID="8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445"
Sep 10 16:06:44.270897 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.270875    2560 scope.go:117] "RemoveContainer" containerID="8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445"
Sep 10 16:06:44.271178 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:44.271158    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445\": container with ID starting with 8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445 not found: ID does not exist" containerID="8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445"
Sep 10 16:06:44.271266 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:44.271188    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445"} err="failed to get container status \"8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445\": rpc error: code = NotFound desc = could not find container \"8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445\": container with ID starting with 8b6316c33f009e0fe05c518da6f0dcc7da3bd33479519abd75475facfa42b445 not found: ID does not exist"
Sep 10 16:06:45.442851 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:45.442826    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:45.585392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:45.585295    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-458dd\" (UniqueName: \"kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd\") pod \"e1040edb-5ac8-4be5-8f98-d2b3c9764302\" (UID: \"e1040edb-5ac8-4be5-8f98-d2b3c9764302\") "
Sep 10 16:06:45.587503 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:45.587474    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd" (OuterVolumeSpecName: "kube-api-access-458dd") pod "e1040edb-5ac8-4be5-8f98-d2b3c9764302" (UID: "e1040edb-5ac8-4be5-8f98-d2b3c9764302"). InnerVolumeSpecName "kube-api-access-458dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:45.686930 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:45.686889    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-458dd\" (UniqueName: \"kubernetes.io/projected/e1040edb-5ac8-4be5-8f98-d2b3c9764302-kube-api-access-458dd\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:46.115806 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.115770    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" path="/var/lib/kubelet/pods/e1040edb-5ac8-4be5-8f98-d2b3c9764302/volumes"
Sep 10 16:06:46.280978 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.280936    2560 generic.go:356] "Generic (PLEG): container finished" podID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" containerID="4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5" exitCode=137
Sep 10 16:06:46.281160 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.280981    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7520-a13bea"
Sep 10 16:06:46.281160 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.281041    2560 scope.go:117] "RemoveContainer" containerID="4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5"
Sep 10 16:06:46.289285 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.289264    2560 scope.go:117] "RemoveContainer" containerID="4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5"
Sep 10 16:06:46.289598 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:46.289581    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5\": container with ID starting with 4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5 not found: ID does not exist" containerID="4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5"
Sep 10 16:06:46.289660 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:46.289607    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5"} err="failed to get container status \"4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5\": rpc error: code = NotFound desc = could not find container \"4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5\": container with ID starting with 4028316f5857fc4c6ac9458282d5a6f5d3cc8b2bdf99a28d0e8ab08eb4b6a2e5 not found: ID does not exist"
Sep 10 16:06:49.465026 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:49.464990    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:06:54.044642 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.044606    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:06:54.045109 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.044942    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" podUID="df5c614d-b993-498c-bc5e-3b88a587e990" containerName="payload-pre-processing" containerID="cri-o://95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d" gracePeriod=30
Sep 10 16:06:54.316287 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.316208    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:06:54.324963 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.324931    2560 generic.go:356] "Generic (PLEG): container finished" podID="df5c614d-b993-498c-bc5e-3b88a587e990" containerID="95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d" exitCode=0
Sep 10 16:06:54.325114 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.324981    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"
Sep 10 16:06:54.325114 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.324987    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" event={"ID":"df5c614d-b993-498c-bc5e-3b88a587e990","Type":"ContainerDied","Data":"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"}
Sep 10 16:06:54.325114 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.325041    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86" event={"ID":"df5c614d-b993-498c-bc5e-3b88a587e990","Type":"ContainerDied","Data":"2330ef12aaf0f78943b3bba502b5e4df958e7b766826af689e108c19cef0f5e9"}
Sep 10 16:06:54.325114 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.325071    2560 scope.go:117] "RemoveContainer" containerID="95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"
Sep 10 16:06:54.333448 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.333428    2560 scope.go:117] "RemoveContainer" containerID="95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"
Sep 10 16:06:54.333743 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:06:54.333721    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d\": container with ID starting with 95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d not found: ID does not exist" containerID="95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"
Sep 10 16:06:54.333811 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.333753    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d"} err="failed to get container status \"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d\": rpc error: code = NotFound desc = could not find container \"95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d\": container with ID starting with 95a09b35410fc086f3e15a820d6cd232274348c7c5339cbb5c8e24916033157d not found: ID does not exist"
Sep 10 16:06:54.466401 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.466363    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wlpvv\" (UniqueName: \"kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv\") pod \"df5c614d-b993-498c-bc5e-3b88a587e990\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") "
Sep 10 16:06:54.466401 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.466407    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume\") pod \"df5c614d-b993-498c-bc5e-3b88a587e990\" (UID: \"df5c614d-b993-498c-bc5e-3b88a587e990\") "
Sep 10 16:06:54.466867 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.466841    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "df5c614d-b993-498c-bc5e-3b88a587e990" (UID: "df5c614d-b993-498c-bc5e-3b88a587e990"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:06:54.468759 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.468730    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv" (OuterVolumeSpecName: "kube-api-access-wlpvv") pod "df5c614d-b993-498c-bc5e-3b88a587e990" (UID: "df5c614d-b993-498c-bc5e-3b88a587e990"). InnerVolumeSpecName "kube-api-access-wlpvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:06:54.568025 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.567920    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wlpvv\" (UniqueName: \"kubernetes.io/projected/df5c614d-b993-498c-bc5e-3b88a587e990-kube-api-access-wlpvv\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:54.568025 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.567956    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/df5c614d-b993-498c-bc5e-3b88a587e990-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:06:54.651549 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.651512    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:06:54.655763 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:54.655731    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-247f70-6c55d6d86b-brx86"]
Sep 10 16:06:56.116035 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:06:56.115999    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df5c614d-b993-498c-bc5e-3b88a587e990" path="/var/lib/kubelet/pods/df5c614d-b993-498c-bc5e-3b88a587e990/volumes"
Sep 10 16:07:01.748687 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.748647    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:07:01.749067 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.748979    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" containerName="test-curl-7520-a13bea"
Sep 10 16:07:01.749067 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.748991    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" containerName="test-curl-7520-a13bea"
Sep 10 16:07:01.749067 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.749000    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df5c614d-b993-498c-bc5e-3b88a587e990" containerName="payload-pre-processing"
Sep 10 16:07:01.749067 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.749006    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="df5c614d-b993-498c-bc5e-3b88a587e990" containerName="payload-pre-processing"
Sep 10 16:07:01.749212 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.749079    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="df5c614d-b993-498c-bc5e-3b88a587e990" containerName="payload-pre-processing"
Sep 10 16:07:01.749212 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.749087    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="e1040edb-5ac8-4be5-8f98-d2b3c9764302" containerName="test-curl-7520-a13bea"
Sep 10 16:07:01.751813 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.751793    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.755567 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.755541    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"kube-root-ca.crt\""
Sep 10 16:07:01.755693 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.755542    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"default-dockercfg-x4gg9\""
Sep 10 16:07:01.755693 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.755542    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"rate-test-model-b52270-kserve-self-signed-certs\""
Sep 10 16:07:01.755693 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.755579    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"openshift-service-ca.crt\""
Sep 10 16:07:01.764024 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.764000    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:07:01.819833 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819794    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.820027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819853    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.820027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819904    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.820027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819933    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.820027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819954    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5txr\" (UniqueName: \"kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.820027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.819971    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920690 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920637    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920690 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920692    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5txr\" (UniqueName: \"kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920713    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920758    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920794    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.920947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.920858    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.921159 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.921083    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.921225 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.921188    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.921283 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.921240    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.923255 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.923223    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.923488 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.923468    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:01.930499 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:01.930470    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5txr\" (UniqueName: \"kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr\") pod \"rate-test-model-b52270-kserve-5484cd7d7-7zmnn\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") " pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:02.063822 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:02.063736    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-b52270\"/\"default-dockercfg-x4gg9\""
Sep 10 16:07:02.071909 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:02.071867    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:02.352190 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:02.352161    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:07:02.354591 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:07:02.354556    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3e169ca9_a0d7_4376_8697_124db3bded7d.slice/crio-2a0a7763e17ba2e210d51f586ac31980cd775fc3fda63501763f5544efd4f624 WatchSource:0}: Error finding container 2a0a7763e17ba2e210d51f586ac31980cd775fc3fda63501763f5544efd4f624: Status 404 returned error can't find the container with id 2a0a7763e17ba2e210d51f586ac31980cd775fc3fda63501763f5544efd4f624
Sep 10 16:07:02.361500 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:02.361465    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" event={"ID":"3e169ca9-a0d7-4376-8697-124db3bded7d","Type":"ContainerStarted","Data":"2a0a7763e17ba2e210d51f586ac31980cd775fc3fda63501763f5544efd4f624"}
Sep 10 16:07:03.367775 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:03.367732    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" event={"ID":"3e169ca9-a0d7-4376-8697-124db3bded7d","Type":"ContainerStarted","Data":"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"}
Sep 10 16:07:03.387237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:03.387175    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" podStartSLOduration=2.38715541 podStartE2EDuration="2.38715541s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:07:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:07:03.386412163 +0000 UTC m=+1501.766359230" watchObservedRunningTime="2026-09-10 16:07:03.38715541 +0000 UTC m=+1501.767102476"
Sep 10 16:07:04.864920 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:04.864880    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:07:12.072047 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.072004    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:12.072538 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.072063    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:12.084674 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.084638    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:12.419333 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.419248    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:07:12.654318 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.654286    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:07:12.654552 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.654526    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" podUID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" containerName="payload-processing" containerID="cri-o://ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975" gracePeriod=30
Sep 10 16:07:12.669697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.669624    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:07:12.670228 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.670194    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" podUID="81f242c9-fde5-4812-b7e9-f3786d8403ee" containerName="payload-pre-processing" containerID="cri-o://923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d" gracePeriod=30
Sep 10 16:07:12.946065 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.946034    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:07:12.948521 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:12.948500    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:07:13.014827 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.014784    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume\") pod \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") "
Sep 10 16:07:13.015010 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.014843    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-k89ml\" (UniqueName: \"kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml\") pod \"81f242c9-fde5-4812-b7e9-f3786d8403ee\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") "
Sep 10 16:07:13.015010 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.014893    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume\") pod \"81f242c9-fde5-4812-b7e9-f3786d8403ee\" (UID: \"81f242c9-fde5-4812-b7e9-f3786d8403ee\") "
Sep 10 16:07:13.015010 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.014918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7fw42\" (UniqueName: \"kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42\") pod \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\" (UID: \"f92f9f06-7a8d-430e-821c-8ef41d89d7e1\") "
Sep 10 16:07:13.015264 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.015232    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f92f9f06-7a8d-430e-821c-8ef41d89d7e1" (UID: "f92f9f06-7a8d-430e-821c-8ef41d89d7e1"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:13.015368 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.015261    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "81f242c9-fde5-4812-b7e9-f3786d8403ee" (UID: "81f242c9-fde5-4812-b7e9-f3786d8403ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:07:13.017261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.017236    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42" (OuterVolumeSpecName: "kube-api-access-7fw42") pod "f92f9f06-7a8d-430e-821c-8ef41d89d7e1" (UID: "f92f9f06-7a8d-430e-821c-8ef41d89d7e1"). InnerVolumeSpecName "kube-api-access-7fw42". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:13.017359 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.017284    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml" (OuterVolumeSpecName: "kube-api-access-k89ml") pod "81f242c9-fde5-4812-b7e9-f3786d8403ee" (UID: "81f242c9-fde5-4812-b7e9-f3786d8403ee"). InnerVolumeSpecName "kube-api-access-k89ml". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:07:13.116368 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.116321    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:07:13.116368 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.116360    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k89ml\" (UniqueName: \"kubernetes.io/projected/81f242c9-fde5-4812-b7e9-f3786d8403ee-kube-api-access-k89ml\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:07:13.116368 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.116369    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/81f242c9-fde5-4812-b7e9-f3786d8403ee-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:07:13.116368 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.116377    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7fw42\" (UniqueName: \"kubernetes.io/projected/f92f9f06-7a8d-430e-821c-8ef41d89d7e1-kube-api-access-7fw42\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:07:13.410899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.410857    2560 generic.go:356] "Generic (PLEG): container finished" podID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" containerID="ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975" exitCode=0
Sep 10 16:07:13.411110 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.410909    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"
Sep 10 16:07:13.411110 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.410948    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" event={"ID":"f92f9f06-7a8d-430e-821c-8ef41d89d7e1","Type":"ContainerDied","Data":"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"}
Sep 10 16:07:13.411110 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.411012    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7" event={"ID":"f92f9f06-7a8d-430e-821c-8ef41d89d7e1","Type":"ContainerDied","Data":"3a2c6bd318868025250888cb498171d125b4914eff9d3a6a86863918140cc972"}
Sep 10 16:07:13.411110 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.411040    2560 scope.go:117] "RemoveContainer" containerID="ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"
Sep 10 16:07:13.412236 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.412145    2560 generic.go:356] "Generic (PLEG): container finished" podID="81f242c9-fde5-4812-b7e9-f3786d8403ee" containerID="923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d" exitCode=0
Sep 10 16:07:13.412236 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.412175    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" event={"ID":"81f242c9-fde5-4812-b7e9-f3786d8403ee","Type":"ContainerDied","Data":"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"}
Sep 10 16:07:13.412236 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.412195    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"
Sep 10 16:07:13.412236 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.412218    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w" event={"ID":"81f242c9-fde5-4812-b7e9-f3786d8403ee","Type":"ContainerDied","Data":"1e3c8682f859db63d3478377e81cdd7ea585fb1760dc174896b691fc68103ee4"}
Sep 10 16:07:13.420811 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.420789    2560 scope.go:117] "RemoveContainer" containerID="ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"
Sep 10 16:07:13.421211 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:07:13.421135    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975\": container with ID starting with ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975 not found: ID does not exist" containerID="ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"
Sep 10 16:07:13.421211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.421175    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975"} err="failed to get container status \"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975\": rpc error: code = NotFound desc = could not find container \"ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975\": container with ID starting with ca01de2a412bd45f62a8488b7844a2186e0f091f46dfd9b5bf5e0570afd55975 not found: ID does not exist"
Sep 10 16:07:13.421211 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.421194    2560 scope.go:117] "RemoveContainer" containerID="923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"
Sep 10 16:07:13.429695 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.429676    2560 scope.go:117] "RemoveContainer" containerID="923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"
Sep 10 16:07:13.429983 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:07:13.429960    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d\": container with ID starting with 923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d not found: ID does not exist" containerID="923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"
Sep 10 16:07:13.430051 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.429991    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d"} err="failed to get container status \"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d\": rpc error: code = NotFound desc = could not find container \"923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d\": container with ID starting with 923d0742ece7ef924e6d0cd85262159a6eef682a40f2ed4d349c429a4aa34f5d not found: ID does not exist"
Sep 10 16:07:13.441181 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.441152    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:07:13.447029 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.447004    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-298d9b-759458c654-2qv7w"]
Sep 10 16:07:13.459840 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.459810    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:07:13.464894 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:13.464870    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-298d9b-79d4765cd7-mw7g7"]
Sep 10 16:07:14.116757 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:14.116728    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="81f242c9-fde5-4812-b7e9-f3786d8403ee" path="/var/lib/kubelet/pods/81f242c9-fde5-4812-b7e9-f3786d8403ee/volumes"
Sep 10 16:07:14.117151 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:14.117064    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" path="/var/lib/kubelet/pods/f92f9f06-7a8d-430e-821c-8ef41d89d7e1/volumes"
Sep 10 16:07:17.759494 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:17.759457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:07:43.966967 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:07:43.966922    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:08:07.960494 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:07.960457    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:08:23.257911 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:23.257873    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:08:27.010864 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:27.010829    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:08:32.183261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:32.183222    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:08:42.030628 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.030577    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" podUID="3e169ca9-a0d7-4376-8697-124db3bded7d" containerName="main" containerID="cri-o://fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896" gracePeriod=45
Sep 10 16:08:42.290189 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.290132    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:08:42.383347 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383316    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383565 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383387    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-j5txr\" (UniqueName: \"kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383565 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383469    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383701 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383599    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home" (OuterVolumeSpecName: "home") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:42.383701 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383608    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383701 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383673    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:42.383701 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383692    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383922 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383729    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm\") pod \"3e169ca9-a0d7-4376-8697-124db3bded7d\" (UID: \"3e169ca9-a0d7-4376-8697-124db3bded7d\") "
Sep 10 16:08:42.383973 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383954    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.383973 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383957    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache" (OuterVolumeSpecName: "model-cache") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:42.384078 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.383974    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.385927 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.385906    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm" (OuterVolumeSpecName: "dshm") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:08:42.385927 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.385916    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:08:42.386074 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.385945    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr" (OuterVolumeSpecName: "kube-api-access-j5txr") pod "3e169ca9-a0d7-4376-8697-124db3bded7d" (UID: "3e169ca9-a0d7-4376-8697-124db3bded7d"). InnerVolumeSpecName "kube-api-access-j5txr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:08:42.484658 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.484621    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5txr\" (UniqueName: \"kubernetes.io/projected/3e169ca9-a0d7-4376-8697-124db3bded7d-kube-api-access-j5txr\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.484658 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.484648    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3e169ca9-a0d7-4376-8697-124db3bded7d-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.484658 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.484659    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.484658 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.484667    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3e169ca9-a0d7-4376-8697-124db3bded7d-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:08:42.795990 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.795961    2560 generic.go:356] "Generic (PLEG): container finished" podID="3e169ca9-a0d7-4376-8697-124db3bded7d" containerID="fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896" exitCode=0
Sep 10 16:08:42.796191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.796015    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"
Sep 10 16:08:42.796191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.796044    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" event={"ID":"3e169ca9-a0d7-4376-8697-124db3bded7d","Type":"ContainerDied","Data":"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"}
Sep 10 16:08:42.796191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.796084    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn" event={"ID":"3e169ca9-a0d7-4376-8697-124db3bded7d","Type":"ContainerDied","Data":"2a0a7763e17ba2e210d51f586ac31980cd775fc3fda63501763f5544efd4f624"}
Sep 10 16:08:42.796191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.796101    2560 scope.go:117] "RemoveContainer" containerID="fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"
Sep 10 16:08:42.805162 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.805143    2560 scope.go:117] "RemoveContainer" containerID="fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"
Sep 10 16:08:42.805510 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:08:42.805487    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896\": container with ID starting with fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896 not found: ID does not exist" containerID="fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"
Sep 10 16:08:42.805585 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.805523    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896"} err="failed to get container status \"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896\": rpc error: code = NotFound desc = could not find container \"fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896\": container with ID starting with fba4f7b2cf8880fe834acebe366c52a67e14be68d21bddbbe1326a5671dbc896 not found: ID does not exist"
Sep 10 16:08:42.817677 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.817654    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:08:42.822016 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:42.821995    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-b52270/rate-test-model-b52270-kserve-5484cd7d7-7zmnn"]
Sep 10 16:08:44.115648 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:44.115604    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3e169ca9-a0d7-4376-8697-124db3bded7d" path="/var/lib/kubelet/pods/3e169ca9-a0d7-4376-8697-124db3bded7d/volumes"
Sep 10 16:08:44.863395 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:08:44.863322    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:09:03.261241 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261203    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261514    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="81f242c9-fde5-4812-b7e9-f3786d8403ee" containerName="payload-pre-processing"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261525    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="81f242c9-fde5-4812-b7e9-f3786d8403ee" containerName="payload-pre-processing"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261541    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3e169ca9-a0d7-4376-8697-124db3bded7d" containerName="main"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261547    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="3e169ca9-a0d7-4376-8697-124db3bded7d" containerName="main"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261555    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" containerName="payload-processing"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261560    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" containerName="payload-processing"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261613    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="3e169ca9-a0d7-4376-8697-124db3bded7d" containerName="main"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261622    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="81f242c9-fde5-4812-b7e9-f3786d8403ee" containerName="payload-pre-processing"
Sep 10 16:09:03.261697 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.261628    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f92f9f06-7a8d-430e-821c-8ef41d89d7e1" containerName="payload-processing"
Sep 10 16:09:03.265480 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.265461    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.270650 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.270623    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-ddc985\""
Sep 10 16:09:03.270823 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.270625    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-ddc985-dockercfg-r5gjk\""
Sep 10 16:09:03.276423 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.276398    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:09:03.336564 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.336518    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.336747 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.336589    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzddb\" (UniqueName: \"kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.365492 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.365457    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:09:03.368574 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.368557    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.382839 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.382809    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:09:03.437738 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.437697    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.437949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.437748    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xxcg7\" (UniqueName: \"kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.437949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.437804    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzddb\" (UniqueName: \"kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.437949 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.437899    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.438516 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.438495    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.448745 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.448720    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzddb\" (UniqueName: \"kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb\") pod \"payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.539181 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.539078    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.539181 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.539146    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xxcg7\" (UniqueName: \"kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.539796 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.539776    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.548740 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.548714    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xxcg7\" (UniqueName: \"kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7\") pod \"payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.575623 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.575590    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:03.678220 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.678174    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:03.848106 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.847911    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:09:03.850706 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:09:03.850680    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf2457a86_d66c_430e_b416_f54236cab64a.slice/crio-a3b6d7ee60d3289d7a5ccd4681deeaba9794438539e2ffe1b036fab5defb5026 WatchSource:0}: Error finding container a3b6d7ee60d3289d7a5ccd4681deeaba9794438539e2ffe1b036fab5defb5026: Status 404 returned error can't find the container with id a3b6d7ee60d3289d7a5ccd4681deeaba9794438539e2ffe1b036fab5defb5026
Sep 10 16:09:03.886837 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.886807    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" event={"ID":"f2457a86-d66c-430e-b416-f54236cab64a","Type":"ContainerStarted","Data":"a3b6d7ee60d3289d7a5ccd4681deeaba9794438539e2ffe1b036fab5defb5026"}
Sep 10 16:09:03.951739 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:03.951705    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:09:03.954736 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:09:03.954709    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc17e468a_b77a_4eb9_816c_4ff29b793f37.slice/crio-0d5f8d19c0958ec2bc41abe81eb49504fa60eac9217324484e90f41818d76ac3 WatchSource:0}: Error finding container 0d5f8d19c0958ec2bc41abe81eb49504fa60eac9217324484e90f41818d76ac3: Status 404 returned error can't find the container with id 0d5f8d19c0958ec2bc41abe81eb49504fa60eac9217324484e90f41818d76ac3
Sep 10 16:09:04.899794 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.899758    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" event={"ID":"f2457a86-d66c-430e-b416-f54236cab64a","Type":"ContainerStarted","Data":"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"}
Sep 10 16:09:04.900241 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.899896    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:04.901382 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.901335    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" event={"ID":"c17e468a-b77a-4eb9-816c-4ff29b793f37","Type":"ContainerStarted","Data":"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"}
Sep 10 16:09:04.901520 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.901403    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:04.901520 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.901425    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" event={"ID":"c17e468a-b77a-4eb9-816c-4ff29b793f37","Type":"ContainerStarted","Data":"0d5f8d19c0958ec2bc41abe81eb49504fa60eac9217324484e90f41818d76ac3"}
Sep 10 16:09:04.919909 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.919849    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" podStartSLOduration=1.919828743 podStartE2EDuration="1.919828743s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:04.917518869 +0000 UTC m=+1623.297465939" watchObservedRunningTime="2026-09-10 16:09:04.919828743 +0000 UTC m=+1623.299775810"
Sep 10 16:09:04.937753 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:04.937692    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" podStartSLOduration=1.9376731600000001 podStartE2EDuration="1.93767316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:04.936081662 +0000 UTC m=+1623.316028790" watchObservedRunningTime="2026-09-10 16:09:04.93767316 +0000 UTC m=+1623.317620229"
Sep 10 16:09:09.170008 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:09.169971    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:09:15.908168 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:15.908131    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:09:15.908645 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:15.908222    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:09:31.301967 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:31.301938    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr_c17e468a-b77a-4eb9-816c-4ff29b793f37/payload-processing/0.log"
Sep 10 16:09:55.171871 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.171831    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:09:55.175574 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.175551    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.178036 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.178016    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-fc8c1e\"/\"default-dockercfg-jw2lt\""
Sep 10 16:09:55.179022 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.178995    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-fc8c1e\"/\"kube-root-ca.crt\""
Sep 10 16:09:55.179124 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.178995    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-fc8c1e\"/\"ipp-route-fc8c1e-kserve-self-signed-certs\""
Sep 10 16:09:55.179124 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.179028    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-fc8c1e\"/\"openshift-service-ca.crt\""
Sep 10 16:09:55.186080 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.186053    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:09:55.270013 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.269970    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.270202 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.270023    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.270202 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.270115    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzcv5\" (UniqueName: \"kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.270202 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.270171    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.270202 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.270188    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.270416 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.270265    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371304 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371265    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzcv5\" (UniqueName: \"kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371373    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371395    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371427    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371468    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371497    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371809 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371783    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371879 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371820    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.371917 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.371892    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.373910 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.373893    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.374076 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.374060    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.381303 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.381283    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzcv5\" (UniqueName: \"kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5\") pod \"ipp-route-fc8c1e-kserve-d4dc769c9-msqzk\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") " pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.487804 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.487760    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:09:55.764940 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:55.764908    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:09:55.767658 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:09:55.767628    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod09af2472_eb75_4536_8122_065034d8593f.slice/crio-6f3baf5f0cc030bd4b512e8500632c3bdbdd463649b7a367a6a9e76d0a021c5b WatchSource:0}: Error finding container 6f3baf5f0cc030bd4b512e8500632c3bdbdd463649b7a367a6a9e76d0a021c5b: Status 404 returned error can't find the container with id 6f3baf5f0cc030bd4b512e8500632c3bdbdd463649b7a367a6a9e76d0a021c5b
Sep 10 16:09:56.116273 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:56.116195    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" event={"ID":"09af2472-eb75-4536-8122-065034d8593f","Type":"ContainerStarted","Data":"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"}
Sep 10 16:09:56.116273 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:56.116230    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" event={"ID":"09af2472-eb75-4536-8122-065034d8593f","Type":"ContainerStarted","Data":"6f3baf5f0cc030bd4b512e8500632c3bdbdd463649b7a367a6a9e76d0a021c5b"}
Sep 10 16:09:56.136501 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:56.136450    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" podStartSLOduration=1.136434018 podStartE2EDuration="1.136434018s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:09:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:09:56.135838166 +0000 UTC m=+1674.515785233" watchObservedRunningTime="2026-09-10 16:09:56.136434018 +0000 UTC m=+1674.516381083"
Sep 10 16:09:59.577506 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:09:59.577462    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:10:05.488612 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:05.488570    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:10:05.489021 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:05.488823    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:10:05.501397 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:05.501371    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:10:06.171392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:06.171357    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:10:15.864871 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:15.864834    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:10:22.468486 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:22.468453    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr_c17e468a-b77a-4eb9-816c-4ff29b793f37/payload-processing/0.log"
Sep 10 16:10:27.927693 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:27.927656    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:10:27.931239 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:27.931216    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:27.933676 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:27.933648    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-65ee1d\""
Sep 10 16:10:27.933676 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:27.933666    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-65ee1d-dockercfg-nsxc9\""
Sep 10 16:10:27.943296 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:27.943268    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:10:28.026285 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.026248    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.026285 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.026287    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hp88s\" (UniqueName: \"kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.127733 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.127699    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.127733 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.127736    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hp88s\" (UniqueName: \"kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.128435 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.128416    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.136658 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.136630    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hp88s\" (UniqueName: \"kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s\") pod \"payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.240631 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.240589    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:28.513428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:28.513399    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:10:28.515844 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:10:28.515815    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0b094c40_ad37_44f5_842e_651ccd055d2e.slice/crio-f2eac4d1af81a1857dfd375a4abb6d2ea3f5c2f800563298f003f665c263d520 WatchSource:0}: Error finding container f2eac4d1af81a1857dfd375a4abb6d2ea3f5c2f800563298f003f665c263d520: Status 404 returned error can't find the container with id f2eac4d1af81a1857dfd375a4abb6d2ea3f5c2f800563298f003f665c263d520
Sep 10 16:10:29.257503 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:29.257463    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" event={"ID":"0b094c40-ad37-44f5-842e-651ccd055d2e","Type":"ContainerStarted","Data":"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"}
Sep 10 16:10:29.257948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:29.257526    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" event={"ID":"0b094c40-ad37-44f5-842e-651ccd055d2e","Type":"ContainerStarted","Data":"f2eac4d1af81a1857dfd375a4abb6d2ea3f5c2f800563298f003f665c263d520"}
Sep 10 16:10:29.257948 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:29.257634    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:29.274743 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:29.274693    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" podStartSLOduration=2.274677362 podStartE2EDuration="2.274677362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:10:27 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:10:29.274408654 +0000 UTC m=+1707.654355719" watchObservedRunningTime="2026-09-10 16:10:29.274677362 +0000 UTC m=+1707.654624427"
Sep 10 16:10:36.071862 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:36.071821    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:10:40.264521 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:40.264492    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:51.577604 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.577566    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"]
Sep 10 16:10:51.580933 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.580914    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:51.583403 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.583374    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:10:51.583527 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.583482    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:10:51.590888 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.590861    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"]
Sep 10 16:10:51.638726 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.638667    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q\" (UID: \"11c0f358-5578-4d9e-a5dc-40ce68681b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:51.739540 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.739499    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q\" (UID: \"11c0f358-5578-4d9e-a5dc-40ce68681b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:51.740121 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.740101    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q\" (UID: \"11c0f358-5578-4d9e-a5dc-40ce68681b0e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:51.892224 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:51.892113    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:52.171043 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:52.171013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"]
Sep 10 16:10:52.173757 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:10:52.173730    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11c0f358_5578_4d9e_a5dc_40ce68681b0e.slice/crio-6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4 WatchSource:0}: Error finding container 6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4: Status 404 returned error can't find the container with id 6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4
Sep 10 16:10:52.358452 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:52.358415    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q" event={"ID":"11c0f358-5578-4d9e-a5dc-40ce68681b0e","Type":"ContainerStarted","Data":"6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4"}
Sep 10 16:10:53.365526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:53.365494    2560 generic.go:356] "Generic (PLEG): container finished" podID="11c0f358-5578-4d9e-a5dc-40ce68681b0e" containerID="87b087e816e3e7606ccbf78dc3329416cc9d5ffb1f18a0e61300d22b5a274ba2" exitCode=0
Sep 10 16:10:53.365912 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:53.365589    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q" event={"ID":"11c0f358-5578-4d9e-a5dc-40ce68681b0e","Type":"ContainerDied","Data":"87b087e816e3e7606ccbf78dc3329416cc9d5ffb1f18a0e61300d22b5a274ba2"}
Sep 10 16:10:54.495311 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:54.495287    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:54.565048 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:54.565008    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca\") pod \"11c0f358-5578-4d9e-a5dc-40ce68681b0e\" (UID: \"11c0f358-5578-4d9e-a5dc-40ce68681b0e\") "
Sep 10 16:10:54.565428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:54.565408    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "11c0f358-5578-4d9e-a5dc-40ce68681b0e" (UID: "11c0f358-5578-4d9e-a5dc-40ce68681b0e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:10:54.666526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:54.666445    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/11c0f358-5578-4d9e-a5dc-40ce68681b0e-maas-api-service-ca\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:10:55.376630 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:55.376597    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"
Sep 10 16:10:55.376815 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:55.376637    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q" event={"ID":"11c0f358-5578-4d9e-a5dc-40ce68681b0e","Type":"ContainerDied","Data":"6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4"}
Sep 10 16:10:55.376815 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:55.376691    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6d8b078ded7a610fb11230f83506e9304503f9668a08e3a6cd884da0ef86c7f4"
Sep 10 16:10:57.844277 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:57.844241    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:10:57.844723 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:57.844535    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" podUID="0b094c40-ad37-44f5-842e-651ccd055d2e" containerName="payload-pre-processing" containerID="cri-o://b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c" gracePeriod=30
Sep 10 16:10:58.013656 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.013616    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"]
Sep 10 16:10:58.017982 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.017935    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-65ee1d-bbw7q"]
Sep 10 16:10:58.098290 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.098230    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:58.118428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.118392    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="11c0f358-5578-4d9e-a5dc-40ce68681b0e" path="/var/lib/kubelet/pods/11c0f358-5578-4d9e-a5dc-40ce68681b0e/volumes"
Sep 10 16:10:58.199672 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.199636    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume\") pod \"0b094c40-ad37-44f5-842e-651ccd055d2e\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") "
Sep 10 16:10:58.199849 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.199712    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hp88s\" (UniqueName: \"kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s\") pod \"0b094c40-ad37-44f5-842e-651ccd055d2e\" (UID: \"0b094c40-ad37-44f5-842e-651ccd055d2e\") "
Sep 10 16:10:58.200039 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.200014    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0b094c40-ad37-44f5-842e-651ccd055d2e" (UID: "0b094c40-ad37-44f5-842e-651ccd055d2e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:10:58.201880 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.201861    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s" (OuterVolumeSpecName: "kube-api-access-hp88s") pod "0b094c40-ad37-44f5-842e-651ccd055d2e" (UID: "0b094c40-ad37-44f5-842e-651ccd055d2e"). InnerVolumeSpecName "kube-api-access-hp88s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:10:58.301086 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.301037    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0b094c40-ad37-44f5-842e-651ccd055d2e-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.301086 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.301079    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hp88s\" (UniqueName: \"kubernetes.io/projected/0b094c40-ad37-44f5-842e-651ccd055d2e-kube-api-access-hp88s\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:10:58.391439 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.391319    2560 generic.go:356] "Generic (PLEG): container finished" podID="0b094c40-ad37-44f5-842e-651ccd055d2e" containerID="b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c" exitCode=0
Sep 10 16:10:58.391439 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.391385    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"
Sep 10 16:10:58.391439 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.391414    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" event={"ID":"0b094c40-ad37-44f5-842e-651ccd055d2e","Type":"ContainerDied","Data":"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"}
Sep 10 16:10:58.391711 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.391473    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p" event={"ID":"0b094c40-ad37-44f5-842e-651ccd055d2e","Type":"ContainerDied","Data":"f2eac4d1af81a1857dfd375a4abb6d2ea3f5c2f800563298f003f665c263d520"}
Sep 10 16:10:58.391711 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.391501    2560 scope.go:117] "RemoveContainer" containerID="b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"
Sep 10 16:10:58.400404 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.400384    2560 scope.go:117] "RemoveContainer" containerID="b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"
Sep 10 16:10:58.400694 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:10:58.400677    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c\": container with ID starting with b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c not found: ID does not exist" containerID="b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"
Sep 10 16:10:58.400742 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.400702    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c"} err="failed to get container status \"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c\": rpc error: code = NotFound desc = could not find container \"b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c\": container with ID starting with b847dfb5b0cd639bc21b4c7e1e8bb12626672bd9e7668b553ad05f4bfcb6865c not found: ID does not exist"
Sep 10 16:10:58.415670 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.415636    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:10:58.422318 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:10:58.422294    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-65ee1d-5d954958b6-cjf4p"]
Sep 10 16:11:00.115691 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:00.115658    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0b094c40-ad37-44f5-842e-651ccd055d2e" path="/var/lib/kubelet/pods/0b094c40-ad37-44f5-842e-651ccd055d2e/volumes"
Sep 10 16:11:02.468262 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:02.468229    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:11:08.460203 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460166    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"]
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460486    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0b094c40-ad37-44f5-842e-651ccd055d2e" containerName="payload-pre-processing"
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460498    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="0b094c40-ad37-44f5-842e-651ccd055d2e" containerName="payload-pre-processing"
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460524    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="11c0f358-5578-4d9e-a5dc-40ce68681b0e" containerName="revoke-keys"
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460530    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="11c0f358-5578-4d9e-a5dc-40ce68681b0e" containerName="revoke-keys"
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460580    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="0b094c40-ad37-44f5-842e-651ccd055d2e" containerName="payload-pre-processing"
Sep 10 16:11:08.460654 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.460586    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="11c0f358-5578-4d9e-a5dc-40ce68681b0e" containerName="revoke-keys"
Sep 10 16:11:08.464458 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.464440    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:08.466931 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.466911    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:11:08.467056 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.466941    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:11:08.471845 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.471819    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"]
Sep 10 16:11:08.591104 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.591061    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml\" (UID: \"4e840645-7f91-46a8-8792-34dd23d6ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:08.691667 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.691623    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml\" (UID: \"4e840645-7f91-46a8-8792-34dd23d6ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:08.692293 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.692270    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml\" (UID: \"4e840645-7f91-46a8-8792-34dd23d6ddc3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:08.775695 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:08.775657    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:09.047039 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:09.047013    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"]
Sep 10 16:11:09.049438 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:11:09.049409    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e840645_7f91_46a8_8792_34dd23d6ddc3.slice/crio-79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d WatchSource:0}: Error finding container 79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d: Status 404 returned error can't find the container with id 79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d
Sep 10 16:11:09.442577 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:09.442529    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml" event={"ID":"4e840645-7f91-46a8-8792-34dd23d6ddc3","Type":"ContainerStarted","Data":"55edc89ddc88ed705ad361679eb4e64b5b000b25a6bba30d84728d8f41e776c1"}
Sep 10 16:11:09.442734 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:09.442585    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml" event={"ID":"4e840645-7f91-46a8-8792-34dd23d6ddc3","Type":"ContainerStarted","Data":"79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d"}
Sep 10 16:11:09.462132 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:09.462075    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml" podStartSLOduration=1.462059877 podStartE2EDuration="1.462059877s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:11:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:11:09.461405101 +0000 UTC m=+1747.841352167" watchObservedRunningTime="2026-09-10 16:11:09.462059877 +0000 UTC m=+1747.842006990"
Sep 10 16:11:10.448625 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:10.448590    2560 generic.go:356] "Generic (PLEG): container finished" podID="4e840645-7f91-46a8-8792-34dd23d6ddc3" containerID="55edc89ddc88ed705ad361679eb4e64b5b000b25a6bba30d84728d8f41e776c1" exitCode=0
Sep 10 16:11:10.448821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:10.448679    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml" event={"ID":"4e840645-7f91-46a8-8792-34dd23d6ddc3","Type":"ContainerDied","Data":"55edc89ddc88ed705ad361679eb4e64b5b000b25a6bba30d84728d8f41e776c1"}
Sep 10 16:11:11.577823 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:11.577800    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:11.719039 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:11.718925    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca\") pod \"4e840645-7f91-46a8-8792-34dd23d6ddc3\" (UID: \"4e840645-7f91-46a8-8792-34dd23d6ddc3\") "
Sep 10 16:11:11.719383 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:11.719332    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4e840645-7f91-46a8-8792-34dd23d6ddc3" (UID: "4e840645-7f91-46a8-8792-34dd23d6ddc3"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:11:11.820464 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:11.820409    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4e840645-7f91-46a8-8792-34dd23d6ddc3-maas-api-service-ca\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:12.459520 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:12.459437    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"
Sep 10 16:11:12.459672 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:12.459436    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml" event={"ID":"4e840645-7f91-46a8-8792-34dd23d6ddc3","Type":"ContainerDied","Data":"79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d"}
Sep 10 16:11:12.459672 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:12.459559    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="79b53c5ae6aa92648d0401487d69d3b14119b0ccf3f309dff59a1e16c2cc754d"
Sep 10 16:11:21.574245 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:21.574205    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:11:24.249410 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:24.249374    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"]
Sep 10 16:11:24.252320 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:24.252297    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-fc8c1e-k7xml"]
Sep 10 16:11:26.120489 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:26.120455    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e840645-7f91-46a8-8792-34dd23d6ddc3" path="/var/lib/kubelet/pods/4e840645-7f91-46a8-8792-34dd23d6ddc3/volumes"
Sep 10 16:11:29.603221 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:29.603186    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:11:35.055328 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:35.055284    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:11:44.623228 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.623162    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" podUID="09af2472-eb75-4536-8122-065034d8593f" containerName="main" containerID="cri-o://5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b" gracePeriod=45
Sep 10 16:11:44.868287 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.868262    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:11:44.991711 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991676    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.991903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991729    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.991903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991752    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mzcv5\" (UniqueName: \"kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.991903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991793    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.991903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991835    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.991903 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.991860    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home\") pod \"09af2472-eb75-4536-8122-065034d8593f\" (UID: \"09af2472-eb75-4536-8122-065034d8593f\") "
Sep 10 16:11:44.992164 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.992025    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:44.992164 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.992108    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache" (OuterVolumeSpecName: "model-cache") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:44.992278 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.992257    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:44.992323 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.992294    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:44.992323 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.992254    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home" (OuterVolumeSpecName: "home") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:44.994161 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.994141    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5" (OuterVolumeSpecName: "kube-api-access-mzcv5") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "kube-api-access-mzcv5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:11:44.994470 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.994445    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm" (OuterVolumeSpecName: "dshm") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:11:44.994470 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:44.994462    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "09af2472-eb75-4536-8122-065034d8593f" (UID: "09af2472-eb75-4536-8122-065034d8593f"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:11:45.092976 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.092939    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:45.092976 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.092969    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzcv5\" (UniqueName: \"kubernetes.io/projected/09af2472-eb75-4536-8122-065034d8593f-kube-api-access-mzcv5\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:45.092976 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.092979    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/09af2472-eb75-4536-8122-065034d8593f-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:45.093216 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.092990    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/09af2472-eb75-4536-8122-065034d8593f-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:11:45.597285 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.597247    2560 generic.go:356] "Generic (PLEG): container finished" podID="09af2472-eb75-4536-8122-065034d8593f" containerID="5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b" exitCode=0
Sep 10 16:11:45.597285 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.597285    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"
Sep 10 16:11:45.597526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.597304    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" event={"ID":"09af2472-eb75-4536-8122-065034d8593f","Type":"ContainerDied","Data":"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"}
Sep 10 16:11:45.597526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.597366    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk" event={"ID":"09af2472-eb75-4536-8122-065034d8593f","Type":"ContainerDied","Data":"6f3baf5f0cc030bd4b512e8500632c3bdbdd463649b7a367a6a9e76d0a021c5b"}
Sep 10 16:11:45.597526 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.597389    2560 scope.go:117] "RemoveContainer" containerID="5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"
Sep 10 16:11:45.606154 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.606137    2560 scope.go:117] "RemoveContainer" containerID="5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"
Sep 10 16:11:45.606498 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:11:45.606478    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b\": container with ID starting with 5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b not found: ID does not exist" containerID="5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"
Sep 10 16:11:45.606543 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.606506    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b"} err="failed to get container status \"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b\": rpc error: code = NotFound desc = could not find container \"5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b\": container with ID starting with 5025d14d985512ea90aa309609113ddbacb43e7ae5b2ed76e20dd79c626d0c5b not found: ID does not exist"
Sep 10 16:11:45.619502 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.619466    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:11:45.625027 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:45.624999    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-fc8c1e/ipp-route-fc8c1e-kserve-d4dc769c9-msqzk"]
Sep 10 16:11:46.115929 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:11:46.115896    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09af2472-eb75-4536-8122-065034d8593f" path="/var/lib/kubelet/pods/09af2472-eb75-4536-8122-065034d8593f/volumes"
Sep 10 16:12:15.232530 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232494    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"]
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232793    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09af2472-eb75-4536-8122-065034d8593f" containerName="main"
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232804    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="09af2472-eb75-4536-8122-065034d8593f" containerName="main"
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232815    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e840645-7f91-46a8-8792-34dd23d6ddc3" containerName="revoke-keys"
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232821    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e840645-7f91-46a8-8792-34dd23d6ddc3" containerName="revoke-keys"
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232884    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="09af2472-eb75-4536-8122-065034d8593f" containerName="main"
Sep 10 16:12:15.232938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.232893    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e840645-7f91-46a8-8792-34dd23d6ddc3" containerName="revoke-keys"
Sep 10 16:12:15.237059 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.237042    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:15.239718 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.239689    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:12:15.239877 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.239745    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:12:15.241951 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.241929    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj\" (UID: \"235ea18f-f866-41e7-8470-cc1c3ed5467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:15.247796 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.247770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"]
Sep 10 16:12:15.342448 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.342408    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj\" (UID: \"235ea18f-f866-41e7-8470-cc1c3ed5467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:15.343038 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.343016    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj\" (UID: \"235ea18f-f866-41e7-8470-cc1c3ed5467d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:15.547198 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.547115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:15.822993 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:15.822915    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"]
Sep 10 16:12:15.825723 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:12:15.825697    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod235ea18f_f866_41e7_8470_cc1c3ed5467d.slice/crio-a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa WatchSource:0}: Error finding container a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa: Status 404 returned error can't find the container with id a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa
Sep 10 16:12:16.739657 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:16.739622    2560 generic.go:356] "Generic (PLEG): container finished" podID="235ea18f-f866-41e7-8470-cc1c3ed5467d" containerID="27b5faf81be9e2bfc2697d5aba5d3386cb3bb14e771ebf4dea65746fe602d745" exitCode=0
Sep 10 16:12:16.740061 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:16.739660    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj" event={"ID":"235ea18f-f866-41e7-8470-cc1c3ed5467d","Type":"ContainerDied","Data":"27b5faf81be9e2bfc2697d5aba5d3386cb3bb14e771ebf4dea65746fe602d745"}
Sep 10 16:12:16.740061 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:16.739725    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj" event={"ID":"235ea18f-f866-41e7-8470-cc1c3ed5467d","Type":"ContainerStarted","Data":"a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa"}
Sep 10 16:12:17.877098 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:17.877070    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:17.959535 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:17.959501    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca\") pod \"235ea18f-f866-41e7-8470-cc1c3ed5467d\" (UID: \"235ea18f-f866-41e7-8470-cc1c3ed5467d\") "
Sep 10 16:12:17.959893 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:17.959866    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "235ea18f-f866-41e7-8470-cc1c3ed5467d" (UID: "235ea18f-f866-41e7-8470-cc1c3ed5467d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:18.060170 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:18.060083    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/235ea18f-f866-41e7-8470-cc1c3ed5467d-maas-api-service-ca\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:18.750665 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:18.750627    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"
Sep 10 16:12:18.750665 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:18.750630    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj" event={"ID":"235ea18f-f866-41e7-8470-cc1c3ed5467d","Type":"ContainerDied","Data":"a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa"}
Sep 10 16:12:18.750878 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:18.750683    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a35bb190a0993f926cc5e383fd6d4fa8b581982b28459de9c13f82b318515afa"
Sep 10 16:12:25.910008 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:25.909970    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:12:25.910684 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:25.910236    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" podUID="c17e468a-b77a-4eb9-816c-4ff29b793f37" containerName="payload-processing" containerID="cri-o://e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715" gracePeriod=30
Sep 10 16:12:25.919605 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:25.919571    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:12:25.919874 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:25.919804    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" podUID="f2457a86-d66c-430e-b416-f54236cab64a" containerName="payload-pre-processing" containerID="cri-o://c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27" gracePeriod=30
Sep 10 16:12:26.189019 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.188992    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:12:26.191590 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.191573    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:12:26.219191 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219160    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume\") pod \"c17e468a-b77a-4eb9-816c-4ff29b793f37\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") "
Sep 10 16:12:26.219391 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219198    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xxcg7\" (UniqueName: \"kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7\") pod \"c17e468a-b77a-4eb9-816c-4ff29b793f37\" (UID: \"c17e468a-b77a-4eb9-816c-4ff29b793f37\") "
Sep 10 16:12:26.219391 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219243    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wzddb\" (UniqueName: \"kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb\") pod \"f2457a86-d66c-430e-b416-f54236cab64a\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") "
Sep 10 16:12:26.219391 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219275    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume\") pod \"f2457a86-d66c-430e-b416-f54236cab64a\" (UID: \"f2457a86-d66c-430e-b416-f54236cab64a\") "
Sep 10 16:12:26.219652 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219536    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c17e468a-b77a-4eb9-816c-4ff29b793f37" (UID: "c17e468a-b77a-4eb9-816c-4ff29b793f37"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:26.219763 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.219699    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f2457a86-d66c-430e-b416-f54236cab64a" (UID: "f2457a86-d66c-430e-b416-f54236cab64a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:26.221606 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.221579    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7" (OuterVolumeSpecName: "kube-api-access-xxcg7") pod "c17e468a-b77a-4eb9-816c-4ff29b793f37" (UID: "c17e468a-b77a-4eb9-816c-4ff29b793f37"). InnerVolumeSpecName "kube-api-access-xxcg7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:26.221686 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.221612    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb" (OuterVolumeSpecName: "kube-api-access-wzddb") pod "f2457a86-d66c-430e-b416-f54236cab64a" (UID: "f2457a86-d66c-430e-b416-f54236cab64a"). InnerVolumeSpecName "kube-api-access-wzddb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:26.227308 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.227287    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"]
Sep 10 16:12:26.231997 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.231974    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-ddc985-9fftj"]
Sep 10 16:12:26.320598 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.320553    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f2457a86-d66c-430e-b416-f54236cab64a-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.320598 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.320586    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c17e468a-b77a-4eb9-816c-4ff29b793f37-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.320598 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.320598    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xxcg7\" (UniqueName: \"kubernetes.io/projected/c17e468a-b77a-4eb9-816c-4ff29b793f37-kube-api-access-xxcg7\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.320598 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.320608    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzddb\" (UniqueName: \"kubernetes.io/projected/f2457a86-d66c-430e-b416-f54236cab64a-kube-api-access-wzddb\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:26.787312 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.787276    2560 generic.go:356] "Generic (PLEG): container finished" podID="f2457a86-d66c-430e-b416-f54236cab64a" containerID="c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27" exitCode=0
Sep 10 16:12:26.787510 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.787330    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"
Sep 10 16:12:26.787510 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.787372    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" event={"ID":"f2457a86-d66c-430e-b416-f54236cab64a","Type":"ContainerDied","Data":"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"}
Sep 10 16:12:26.787510 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.787423    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr" event={"ID":"f2457a86-d66c-430e-b416-f54236cab64a","Type":"ContainerDied","Data":"a3b6d7ee60d3289d7a5ccd4681deeaba9794438539e2ffe1b036fab5defb5026"}
Sep 10 16:12:26.787510 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.787453    2560 scope.go:117] "RemoveContainer" containerID="c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"
Sep 10 16:12:26.788787 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.788765    2560 generic.go:356] "Generic (PLEG): container finished" podID="c17e468a-b77a-4eb9-816c-4ff29b793f37" containerID="e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715" exitCode=0
Sep 10 16:12:26.788879 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.788809    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"
Sep 10 16:12:26.788879 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.788856    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" event={"ID":"c17e468a-b77a-4eb9-816c-4ff29b793f37","Type":"ContainerDied","Data":"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"}
Sep 10 16:12:26.789218 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.788897    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr" event={"ID":"c17e468a-b77a-4eb9-816c-4ff29b793f37","Type":"ContainerDied","Data":"0d5f8d19c0958ec2bc41abe81eb49504fa60eac9217324484e90f41818d76ac3"}
Sep 10 16:12:26.796426 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.796408    2560 scope.go:117] "RemoveContainer" containerID="c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"
Sep 10 16:12:26.796765 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:12:26.796738    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27\": container with ID starting with c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27 not found: ID does not exist" containerID="c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"
Sep 10 16:12:26.796821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.796783    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27"} err="failed to get container status \"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27\": rpc error: code = NotFound desc = could not find container \"c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27\": container with ID starting with c5dcd4811931b97bfef8e2f88a77aa771aac32e4032c7e04becf826a54976e27 not found: ID does not exist"
Sep 10 16:12:26.796821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.796807    2560 scope.go:117] "RemoveContainer" containerID="e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"
Sep 10 16:12:26.804819 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.804800    2560 scope.go:117] "RemoveContainer" containerID="e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"
Sep 10 16:12:26.805090 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:12:26.805071    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715\": container with ID starting with e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715 not found: ID does not exist" containerID="e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"
Sep 10 16:12:26.805140 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.805101    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715"} err="failed to get container status \"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715\": rpc error: code = NotFound desc = could not find container \"e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715\": container with ID starting with e6443ad03359b4d35922794fcd157ceabc42a6422f3bbdf7b22d8711822d5715 not found: ID does not exist"
Sep 10 16:12:26.820875 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.820844    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:12:26.826020 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.825997    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-ddc985-86c7b765b5-jwdsr"]
Sep 10 16:12:26.841098 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.841069    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:12:26.846374 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:26.846249    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-ddc985-c7b9587bb-m5kgr"]
Sep 10 16:12:28.115668 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:28.115636    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="235ea18f-f866-41e7-8470-cc1c3ed5467d" path="/var/lib/kubelet/pods/235ea18f-f866-41e7-8470-cc1c3ed5467d/volumes"
Sep 10 16:12:28.116030 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:28.115966    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c17e468a-b77a-4eb9-816c-4ff29b793f37" path="/var/lib/kubelet/pods/c17e468a-b77a-4eb9-816c-4ff29b793f37/volumes"
Sep 10 16:12:28.116263 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:28.116250    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f2457a86-d66c-430e-b416-f54236cab64a" path="/var/lib/kubelet/pods/f2457a86-d66c-430e-b416-f54236cab64a/volumes"
Sep 10 16:12:29.360066 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:29.360025    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:12:33.570979 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.570938    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"]
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571235    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f2457a86-d66c-430e-b416-f54236cab64a" containerName="payload-pre-processing"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571246    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="f2457a86-d66c-430e-b416-f54236cab64a" containerName="payload-pre-processing"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571258    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c17e468a-b77a-4eb9-816c-4ff29b793f37" containerName="payload-processing"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571263    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c17e468a-b77a-4eb9-816c-4ff29b793f37" containerName="payload-processing"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571274    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="235ea18f-f866-41e7-8470-cc1c3ed5467d" containerName="revoke-keys"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571279    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="235ea18f-f866-41e7-8470-cc1c3ed5467d" containerName="revoke-keys"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571328    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="235ea18f-f866-41e7-8470-cc1c3ed5467d" containerName="revoke-keys"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571335    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c17e468a-b77a-4eb9-816c-4ff29b793f37" containerName="payload-processing"
Sep 10 16:12:33.571406 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.571363    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="f2457a86-d66c-430e-b416-f54236cab64a" containerName="payload-pre-processing"
Sep 10 16:12:33.574219 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.574201    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:33.576769 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.576746    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-2wghg\""
Sep 10 16:12:33.576899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.576768    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 10 16:12:33.586520 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.586491    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"]
Sep 10 16:12:33.678974 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.678938    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp\" (UID: \"c57f9abc-28c3-4650-96ed-499994589d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:33.779755 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.779718    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp\" (UID: \"c57f9abc-28c3-4650-96ed-499994589d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:33.780387 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.780369    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp\" (UID: \"c57f9abc-28c3-4650-96ed-499994589d6b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:33.884043 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:33.883953    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:34.162225 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:34.162139    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"]
Sep 10 16:12:34.169824 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:12:34.169793    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc57f9abc_28c3_4650_96ed_499994589d6b.slice/crio-7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8 WatchSource:0}: Error finding container 7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8: Status 404 returned error can't find the container with id 7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8
Sep 10 16:12:34.827866 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:34.827839    2560 generic.go:356] "Generic (PLEG): container finished" podID="c57f9abc-28c3-4650-96ed-499994589d6b" containerID="29fac068013794b6f5442b1e393db945789ad9d95c246e9ba97a619ad0a4229f" exitCode=0
Sep 10 16:12:34.828237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:34.827921    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp" event={"ID":"c57f9abc-28c3-4650-96ed-499994589d6b","Type":"ContainerDied","Data":"29fac068013794b6f5442b1e393db945789ad9d95c246e9ba97a619ad0a4229f"}
Sep 10 16:12:34.828237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:34.828024    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp" event={"ID":"c57f9abc-28c3-4650-96ed-499994589d6b","Type":"ContainerStarted","Data":"7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8"}
Sep 10 16:12:35.958178 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:35.958148    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:35.998490 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:35.998455    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca\") pod \"c57f9abc-28c3-4650-96ed-499994589d6b\" (UID: \"c57f9abc-28c3-4650-96ed-499994589d6b\") "
Sep 10 16:12:35.998788 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:35.998761    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c57f9abc-28c3-4650-96ed-499994589d6b" (UID: "c57f9abc-28c3-4650-96ed-499994589d6b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:36.099968 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:36.099879    2560 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c57f9abc-28c3-4650-96ed-499994589d6b-maas-api-service-ca\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:36.837052 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:36.837019    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"
Sep 10 16:12:36.837052 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:36.837031    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp" event={"ID":"c57f9abc-28c3-4650-96ed-499994589d6b","Type":"ContainerDied","Data":"7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8"}
Sep 10 16:12:36.837265 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:36.837072    2560 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7bd67c9e7c226ddeb98527423997f420e3a08e9d4469046fe138c68219a299a8"
Sep 10 16:12:44.308893 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.308853    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:12:44.309474 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.309164    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" podUID="954756ba-e36a-452e-a5b5-e889c909ace9" containerName="payload-processing" containerID="cri-o://e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77" gracePeriod=30
Sep 10 16:12:44.558391 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.558366    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:12:44.643407 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.643316    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"]
Sep 10 16:12:44.651925 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.651895    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-b52270-xpjhp"]
Sep 10 16:12:44.666806 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.666774    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume\") pod \"954756ba-e36a-452e-a5b5-e889c909ace9\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") "
Sep 10 16:12:44.666951 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.666918    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6kqsz\" (UniqueName: \"kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz\") pod \"954756ba-e36a-452e-a5b5-e889c909ace9\" (UID: \"954756ba-e36a-452e-a5b5-e889c909ace9\") "
Sep 10 16:12:44.667161 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.667136    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "954756ba-e36a-452e-a5b5-e889c909ace9" (UID: "954756ba-e36a-452e-a5b5-e889c909ace9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:12:44.669092 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.669062    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz" (OuterVolumeSpecName: "kube-api-access-6kqsz") pod "954756ba-e36a-452e-a5b5-e889c909ace9" (UID: "954756ba-e36a-452e-a5b5-e889c909ace9"). InnerVolumeSpecName "kube-api-access-6kqsz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:12:44.767756 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.767714    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kqsz\" (UniqueName: \"kubernetes.io/projected/954756ba-e36a-452e-a5b5-e889c909ace9-kube-api-access-6kqsz\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:44.767756 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.767748    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/954756ba-e36a-452e-a5b5-e889c909ace9-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:12:44.881831 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.881792    2560 generic.go:356] "Generic (PLEG): container finished" podID="954756ba-e36a-452e-a5b5-e889c909ace9" containerID="e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77" exitCode=0
Sep 10 16:12:44.881999 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.881839    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"
Sep 10 16:12:44.881999 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.881875    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" event={"ID":"954756ba-e36a-452e-a5b5-e889c909ace9","Type":"ContainerDied","Data":"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"}
Sep 10 16:12:44.881999 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.881932    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv" event={"ID":"954756ba-e36a-452e-a5b5-e889c909ace9","Type":"ContainerDied","Data":"f3abe47bb568511b030fbacab7240e38843a0c8fe0ce03a01ea0e19b8abc53a4"}
Sep 10 16:12:44.881999 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.881953    2560 scope.go:117] "RemoveContainer" containerID="e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"
Sep 10 16:12:44.890482 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.890462    2560 scope.go:117] "RemoveContainer" containerID="e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"
Sep 10 16:12:44.890803 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:12:44.890783    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77\": container with ID starting with e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77 not found: ID does not exist" containerID="e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"
Sep 10 16:12:44.890851 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.890823    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77"} err="failed to get container status \"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77\": rpc error: code = NotFound desc = could not find container \"e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77\": container with ID starting with e295661a5d8e4ee6cb3442d260eacef415f9e42d4c698ab9d776e796481a8d77 not found: ID does not exist"
Sep 10 16:12:44.920243 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.920155    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:12:44.930216 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:44.930182    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-b52270-7cbd87c7cf-wcnfv"]
Sep 10 16:12:46.114933 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:46.114900    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="954756ba-e36a-452e-a5b5-e889c909ace9" path="/var/lib/kubelet/pods/954756ba-e36a-452e-a5b5-e889c909ace9/volumes"
Sep 10 16:12:46.115325 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:46.115253    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c57f9abc-28c3-4650-96ed-499994589d6b" path="/var/lib/kubelet/pods/c57f9abc-28c3-4650-96ed-499994589d6b/volumes"
Sep 10 16:12:47.325236 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:12:47.325198    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:13:04.762149 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:04.762111    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:13:04.762696 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:04.762378    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" podUID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" containerName="payload-processing" containerID="cri-o://1b30f9022b0d245cbc8e6c25738a210fa3a03718539985f648b044e96cbbd1f9" gracePeriod=30
Sep 10 16:13:04.970276 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:04.970233    2560 generic.go:356] "Generic (PLEG): container finished" podID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" containerID="1b30f9022b0d245cbc8e6c25738a210fa3a03718539985f648b044e96cbbd1f9" exitCode=0
Sep 10 16:13:04.970452 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:04.970279    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" event={"ID":"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9","Type":"ContainerDied","Data":"1b30f9022b0d245cbc8e6c25738a210fa3a03718539985f648b044e96cbbd1f9"}
Sep 10 16:13:05.015104 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.015042    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:13:05.138754 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.138713    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume\") pod \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") "
Sep 10 16:13:05.138950 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.138785    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x6hmz\" (UniqueName: \"kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz\") pod \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\" (UID: \"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9\") "
Sep 10 16:13:05.139121 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.139089    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" (UID: "2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 10 16:13:05.141130 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.141104    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz" (OuterVolumeSpecName: "kube-api-access-x6hmz") pod "2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" (UID: "2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9"). InnerVolumeSpecName "kube-api-access-x6hmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:13:05.239512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.239472    2560 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-plugins-config-volume\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:13:05.239512 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.239502    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x6hmz\" (UniqueName: \"kubernetes.io/projected/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9-kube-api-access-x6hmz\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:13:05.975840 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.975806    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"
Sep 10 16:13:05.976261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.975839    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf" event={"ID":"2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9","Type":"ContainerDied","Data":"4186e7ffc5d0e5393e85c5df1147cc81567b77226857dc02e60174be20a8b43f"}
Sep 10 16:13:05.976261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:05.975901    2560 scope.go:117] "RemoveContainer" containerID="1b30f9022b0d245cbc8e6c25738a210fa3a03718539985f648b044e96cbbd1f9"
Sep 10 16:13:06.007821 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:06.007774    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:13:06.011968 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:06.011942    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-108dbb-6c4d78fccf-rrkrf"]
Sep 10 16:13:06.115106 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:06.115074    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" path="/var/lib/kubelet/pods/2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9/volumes"
Sep 10 16:13:07.471817 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:07.471780    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:13:16.872141 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:13:16.872101    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:14:37.759277 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:14:37.759238    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:03.064576 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:03.064490    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:11.064988 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:11.064944    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:18.362897 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:18.362857    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:25.358806 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:25.358770    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:39.468563 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:39.468520    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:15:49.862067 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:15:49.862030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:17:02.906484 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:02.906447    2560 scope.go:117] "RemoveContainer" containerID="87b087e816e3e7606ccbf78dc3329416cc9d5ffb1f18a0e61300d22b5a274ba2"
Sep 10 16:17:18.664195 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:18.664149    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:17:27.157071 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:27.157030    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:17:33.363711 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:33.363672    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:17:49.046694 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.046652    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.046972    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c57f9abc-28c3-4650-96ed-499994589d6b" containerName="revoke-keys"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.046986    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="c57f9abc-28c3-4650-96ed-499994589d6b" containerName="revoke-keys"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047001    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="954756ba-e36a-452e-a5b5-e889c909ace9" containerName="payload-processing"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047007    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="954756ba-e36a-452e-a5b5-e889c909ace9" containerName="payload-processing"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047019    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" containerName="payload-processing"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047025    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" containerName="payload-processing"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047075    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="954756ba-e36a-452e-a5b5-e889c909ace9" containerName="payload-processing"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047085    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="c57f9abc-28c3-4650-96ed-499994589d6b" containerName="revoke-keys"
Sep 10 16:17:49.049097 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.047090    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="2f8a8a7e-abfd-44c2-9e3a-2cb387765ca9" containerName="payload-processing"
Sep 10 16:17:49.049911 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.049892    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.054470 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.054451    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-44b3e5ce-kserve-self-signed-certs\""
Sep 10 16:17:49.069616 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.069583    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:17:49.122833 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.122797    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.122833 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.122834    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.123095 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.122863    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dj6jj\" (UniqueName: \"kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.123095 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.122936    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.123095 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.122997    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.123095 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.123027    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223611 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223569    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223624    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223645    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223687    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223715    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.223820 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.223783    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dj6jj\" (UniqueName: \"kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.224155 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.224126    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.224330 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.224304    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.224330 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.224280    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.226112 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.226093    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.226280 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.226262    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.235033 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.235014    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dj6jj\" (UniqueName: \"kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj\") pod \"e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") " pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.358743 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.358641    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:49.631808 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:49.631729    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:17:49.635309 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:17:49.635282    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda015678a_0b5c_436a_86ca_e555daafe34b.slice/crio-2170250d5662d6770d0ae1fd09a75202e1cc22a70f186aa155cd94047c9ee3be WatchSource:0}: Error finding container 2170250d5662d6770d0ae1fd09a75202e1cc22a70f186aa155cd94047c9ee3be: Status 404 returned error can't find the container with id 2170250d5662d6770d0ae1fd09a75202e1cc22a70f186aa155cd94047c9ee3be
Sep 10 16:17:50.182155 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:50.182116    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" event={"ID":"a015678a-0b5c-436a-86ca-e555daafe34b","Type":"ContainerStarted","Data":"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"}
Sep 10 16:17:50.182575 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:50.182167    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" event={"ID":"a015678a-0b5c-436a-86ca-e555daafe34b","Type":"ContainerStarted","Data":"2170250d5662d6770d0ae1fd09a75202e1cc22a70f186aa155cd94047c9ee3be"}
Sep 10 16:17:50.200872 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:50.200824    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" podStartSLOduration=1.2008073160000001 podStartE2EDuration="1.200807316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:17:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:17:50.200098124 +0000 UTC m=+2148.580045190" watchObservedRunningTime="2026-09-10 16:17:50.200807316 +0000 UTC m=+2148.580754382"
Sep 10 16:17:51.465007 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:51.464967    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:17:59.358954 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:59.358863    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:59.358954 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:59.358909    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:17:59.371414 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:17:59.371376    2560 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:18:00.239359 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:00.239312    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:18:02.925885 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:02.925856    2560 scope.go:117] "RemoveContainer" containerID="55edc89ddc88ed705ad361679eb4e64b5b000b25a6bba30d84728d8f41e776c1"
Sep 10 16:18:04.504049 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:04.504010    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:18:24.028264 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:24.028224    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:18:26.392362 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:26.392308    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:18:35.572535 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:35.572493    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:18:39.047899 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.047851    2560 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" podUID="a015678a-0b5c-436a-86ca-e555daafe34b" containerName="main" containerID="cri-o://c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268" gracePeriod=45
Sep 10 16:18:39.289617 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.289591    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:18:39.344838 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344746    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dj6jj\" (UniqueName: \"kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.344838 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344797    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.344838 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344821    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.345141 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344849    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.345141 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344885    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.345141 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.344914    2560 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache\") pod \"a015678a-0b5c-436a-86ca-e555daafe34b\" (UID: \"a015678a-0b5c-436a-86ca-e555daafe34b\") "
Sep 10 16:18:39.345141 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.345114    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home" (OuterVolumeSpecName: "home") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:39.345392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.345132    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:39.345392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.345223    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache" (OuterVolumeSpecName: "model-cache") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:39.345392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.345256    2560 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-home\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:39.345392 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.345276    2560 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-tmp-dir\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:39.347150 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.347113    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm" (OuterVolumeSpecName: "dshm") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 10 16:18:39.347269 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.347151    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 10 16:18:39.347269 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.347169    2560 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj" (OuterVolumeSpecName: "kube-api-access-dj6jj") pod "a015678a-0b5c-436a-86ca-e555daafe34b" (UID: "a015678a-0b5c-436a-86ca-e555daafe34b"). InnerVolumeSpecName "kube-api-access-dj6jj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 10 16:18:39.391247 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.391213    2560 generic.go:356] "Generic (PLEG): container finished" podID="a015678a-0b5c-436a-86ca-e555daafe34b" containerID="c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268" exitCode=0
Sep 10 16:18:39.391455 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.391258    2560 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"
Sep 10 16:18:39.391455 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.391298    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" event={"ID":"a015678a-0b5c-436a-86ca-e555daafe34b","Type":"ContainerDied","Data":"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"}
Sep 10 16:18:39.391455 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.391369    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x" event={"ID":"a015678a-0b5c-436a-86ca-e555daafe34b","Type":"ContainerDied","Data":"2170250d5662d6770d0ae1fd09a75202e1cc22a70f186aa155cd94047c9ee3be"}
Sep 10 16:18:39.391455 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.391394    2560 scope.go:117] "RemoveContainer" containerID="c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"
Sep 10 16:18:39.402192 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.402173    2560 scope.go:117] "RemoveContainer" containerID="c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"
Sep 10 16:18:39.402505 ip-10-0-138-241 kubenswrapper[2560]: E0910 16:18:39.402486    2560 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268\": container with ID starting with c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268 not found: ID does not exist" containerID="c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"
Sep 10 16:18:39.402572 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.402517    2560 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268"} err="failed to get container status \"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268\": rpc error: code = NotFound desc = could not find container \"c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268\": container with ID starting with c5eca3dab300865d8b213905a8e24362ce5a6e239c137f35f6bcf9e0c9502268 not found: ID does not exist"
Sep 10 16:18:39.414608 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.414577    2560 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:18:39.418537 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.418511    2560 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-44b3e5ce-kserve-6587f56cd7-s2j9x"]
Sep 10 16:18:39.446395 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.446361    2560 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dj6jj\" (UniqueName: \"kubernetes.io/projected/a015678a-0b5c-436a-86ca-e555daafe34b-kube-api-access-dj6jj\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:39.446395 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.446395    2560 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a015678a-0b5c-436a-86ca-e555daafe34b-tls-certs\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:39.446606 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.446410    2560 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-dshm\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:39.446606 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:39.446420    2560 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a015678a-0b5c-436a-86ca-e555daafe34b-model-cache\") on node \"ip-10-0-138-241.ec2.internal\" DevicePath \"\""
Sep 10 16:18:40.115966 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:40.115934    2560 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a015678a-0b5c-436a-86ca-e555daafe34b" path="/var/lib/kubelet/pods/a015678a-0b5c-436a-86ca-e555daafe34b/volumes"
Sep 10 16:18:44.859187 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:44.859152    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:18:52.865315 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:52.865281    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:18:59.358324 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:18:59.358287    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:19:02.943431 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:02.943399    2560 scope.go:117] "RemoveContainer" containerID="27b5faf81be9e2bfc2697d5aba5d3386cb3bb14e771ebf4dea65746fe602d745"
Sep 10 16:19:02.951244 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:02.951225    2560 scope.go:117] "RemoveContainer" containerID="29fac068013794b6f5442b1e393db945789ad9d95c246e9ba97a619ad0a4229f"
Sep 10 16:19:09.061583 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:09.061546    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:19:14.763649 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:14.763610    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-jwkr8"]
Sep 10 16:19:17.994574 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:17.994530    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6675789b4c-x59x5_fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd/authorino/0.log"
Sep 10 16:19:24.124174 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:24.124099    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf_1e9c74bf-07f2-429e-9e3e-968e6dffd734/copy-manifests/0.log"
Sep 10 16:19:24.135373 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:24.135325    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf_1e9c74bf-07f2-429e-9e3e-968e6dffd734/manager/0.log"
Sep 10 16:19:25.847234 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:25.847197    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6675789b4c-x59x5_fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd/authorino/0.log"
Sep 10 16:19:26.190186 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:26.190098    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hr9gt_21229c4c-dace-4cc7-b9a7-ab505624b53f/kuadrant-console-plugin/0.log"
Sep 10 16:19:26.419656 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:26.419619    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jwkr8_8b44eee6-bad7-4137-b352-09830839ae26/limitador/0.log"
Sep 10 16:19:27.095164 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:27.095134    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-dmqnm_740e90ae-dfe3-4404-9ab5-789b28590415/kube-auth-proxy/0.log"
Sep 10 16:19:28.145499 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:28.145472    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-sptfl_7b97fd34-f75d-41d4-aedb-4e18fff4a1fd/main/0.log"
Sep 10 16:19:28.479139 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:28.479084    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68f2stdn_ad76661f-6513-4aa4-852d-42b368b67519/main/0.log"
Sep 10 16:19:28.703205 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:28.703170    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-w5vkc_3f3af9f0-fc25-4bba-b71f-d830336ec488/main/0.log"
Sep 10 16:19:33.704830 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.704789    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2t5gt/must-gather-f4vph"]
Sep 10 16:19:33.705237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.705098    2560 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a015678a-0b5c-436a-86ca-e555daafe34b" containerName="main"
Sep 10 16:19:33.705237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.705109    2560 state_mem.go:107] "Deleted CPUSet assignment" podUID="a015678a-0b5c-436a-86ca-e555daafe34b" containerName="main"
Sep 10 16:19:33.705237 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.705160    2560 memory_manager.go:368] "RemoveStaleState removing state" podUID="a015678a-0b5c-436a-86ca-e555daafe34b" containerName="main"
Sep 10 16:19:33.708617 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.708597    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.711177 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.711152    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"kube-root-ca.crt\""
Sep 10 16:19:33.711290 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.711260    2560 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-2t5gt\"/\"openshift-service-ca.crt\""
Sep 10 16:19:33.712256 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.712243    2560 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-2t5gt\"/\"default-dockercfg-xmrs6\""
Sep 10 16:19:33.718152 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.718128    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/must-gather-f4vph"]
Sep 10 16:19:33.788207 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.788165    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ded661-6ca3-4b0e-bacc-bb92e12329b7-must-gather-output\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.788428 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.788224    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gcfcx\" (UniqueName: \"kubernetes.io/projected/94ded661-6ca3-4b0e-bacc-bb92e12329b7-kube-api-access-gcfcx\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.888624 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.888586    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ded661-6ca3-4b0e-bacc-bb92e12329b7-must-gather-output\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.888807 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.888650    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gcfcx\" (UniqueName: \"kubernetes.io/projected/94ded661-6ca3-4b0e-bacc-bb92e12329b7-kube-api-access-gcfcx\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.889000 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.888972    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/94ded661-6ca3-4b0e-bacc-bb92e12329b7-must-gather-output\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:33.897933 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:33.897901    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gcfcx\" (UniqueName: \"kubernetes.io/projected/94ded661-6ca3-4b0e-bacc-bb92e12329b7-kube-api-access-gcfcx\") pod \"must-gather-f4vph\" (UID: \"94ded661-6ca3-4b0e-bacc-bb92e12329b7\") " pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:34.018588 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:34.018556    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/must-gather-f4vph"
Sep 10 16:19:34.297453 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:34.297375    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/must-gather-f4vph"]
Sep 10 16:19:34.300704 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:19:34.300673    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod94ded661_6ca3_4b0e_bacc_bb92e12329b7.slice/crio-edfe60c7dea4efd80860fb44aab2085a5141eea5a318e4b48823ca1d29f2c397 WatchSource:0}: Error finding container edfe60c7dea4efd80860fb44aab2085a5141eea5a318e4b48823ca1d29f2c397: Status 404 returned error can't find the container with id edfe60c7dea4efd80860fb44aab2085a5141eea5a318e4b48823ca1d29f2c397
Sep 10 16:19:34.302638 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:34.302622    2560 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 10 16:19:34.620144 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:34.620057    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/must-gather-f4vph" event={"ID":"94ded661-6ca3-4b0e-bacc-bb92e12329b7","Type":"ContainerStarted","Data":"edfe60c7dea4efd80860fb44aab2085a5141eea5a318e4b48823ca1d29f2c397"}
Sep 10 16:19:35.631080 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:35.630269    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/must-gather-f4vph" event={"ID":"94ded661-6ca3-4b0e-bacc-bb92e12329b7","Type":"ContainerStarted","Data":"d4995c1035f0933e316b75052a5e47d171be390bf9fd9b3c5c2a615b8d727cb9"}
Sep 10 16:19:35.631080 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:35.630321    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/must-gather-f4vph" event={"ID":"94ded661-6ca3-4b0e-bacc-bb92e12329b7","Type":"ContainerStarted","Data":"ea64ab1b185c0dc636f68ea3d3085e2799229edcb599ec78b6003347a18f2365"}
Sep 10 16:19:35.647531 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:35.647468    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2t5gt/must-gather-f4vph" podStartSLOduration=1.929627907 podStartE2EDuration="2.647444569s" totalImagesPullingTime="717.816662ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-10 16:19:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:35.647039448 +0000 UTC m=+2254.026986515" watchObservedRunningTime="2026-09-10 16:19:35.647444569 +0000 UTC m=+2254.027391638"
Sep 10 16:19:36.664356 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:36.664314    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-drwxg_c7d4a7f5-53a5-49e7-85c6-4f2c375ae688/global-pull-secret-syncer/0.log"
Sep 10 16:19:36.878791 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:36.878747    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-zjv5t_16946fc3-49ae-4ce0-a1ca-a8e4b7c9b752/konnectivity-agent/0.log"
Sep 10 16:19:36.932126 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:36.932046    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-138-241.ec2.internal_17ace8c758eb0a5cfdc6b4f1e8d25f1a/haproxy/0.log"
Sep 10 16:19:41.465664 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:41.465619    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6675789b4c-x59x5_fb8faa6f-e3d5-4a8d-be2b-99d4d6ece9fd/authorino/0.log"
Sep 10 16:19:41.591169 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:41.591143    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-fbdd4f445-hr9gt_21229c4c-dace-4cc7-b9a7-ab505624b53f/kuadrant-console-plugin/0.log"
Sep 10 16:19:41.678698 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:41.678672    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-jwkr8_8b44eee6-bad7-4137-b352-09830839ae26/limitador/0.log"
Sep 10 16:19:44.040108 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:44.040077    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwvsj_8d2dc261-81af-4962-a13e-cdb30cc2572e/node-exporter/0.log"
Sep 10 16:19:44.080599 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:44.080563    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwvsj_8d2dc261-81af-4962-a13e-cdb30cc2572e/kube-rbac-proxy/0.log"
Sep 10 16:19:44.130177 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:44.130147    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-bwvsj_8d2dc261-81af-4962-a13e-cdb30cc2572e/init-textfile/0.log"
Sep 10 16:19:46.421386 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:46.421324    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-79977f9844-v48w7_7b998e29-3193-4a9b-9644-944c0f55b9b3/networking-console-plugin/0.log"
Sep 10 16:19:47.017371 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.017318    2560 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"]
Sep 10 16:19:47.025151 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.025115    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.033947 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.033919    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"]
Sep 10 16:19:47.129922 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.129887    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-proc\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.130127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.129953    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-lib-modules\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.130127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.129984    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xcfzv\" (UniqueName: \"kubernetes.io/projected/d089cb40-086e-44b8-b08a-2ec9605fc832-kube-api-access-xcfzv\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.130127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.130047    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-sys\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.130127 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.130080    2560 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-podres\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231074 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231031    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-proc\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231074 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231081    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-lib-modules\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231366 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231107    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xcfzv\" (UniqueName: \"kubernetes.io/projected/d089cb40-086e-44b8-b08a-2ec9605fc832-kube-api-access-xcfzv\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231366 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231191    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-sys\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231366 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231271    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-lib-modules\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231548 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231410    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-sys\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231624 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231599    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-proc\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231698 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231621    2560 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-podres\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.231775 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.231751    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d089cb40-086e-44b8-b08a-2ec9605fc832-podres\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.244714 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.244683    2560 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xcfzv\" (UniqueName: \"kubernetes.io/projected/d089cb40-086e-44b8-b08a-2ec9605fc832-kube-api-access-xcfzv\") pod \"perf-node-gather-daemonset-ch9mf\" (UID: \"d089cb40-086e-44b8-b08a-2ec9605fc832\") " pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.342938 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.342851    2560 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:47.635557 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.635475    2560 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"]
Sep 10 16:19:47.640171 ip-10-0-138-241 kubenswrapper[2560]: W0910 16:19:47.640129    2560 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd089cb40_086e_44b8_b08a_2ec9605fc832.slice/crio-e8be97637fc0c59a55366bd03948a64e873a42c7b2013dce5a99b9347d78c3ca WatchSource:0}: Error finding container e8be97637fc0c59a55366bd03948a64e873a42c7b2013dce5a99b9347d78c3ca: Status 404 returned error can't find the container with id e8be97637fc0c59a55366bd03948a64e873a42c7b2013dce5a99b9347d78c3ca
Sep 10 16:19:47.715632 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:47.715557    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf" event={"ID":"d089cb40-086e-44b8-b08a-2ec9605fc832","Type":"ContainerStarted","Data":"e8be97637fc0c59a55366bd03948a64e873a42c7b2013dce5a99b9347d78c3ca"}
Sep 10 16:19:48.722921 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:48.722884    2560 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf" event={"ID":"d089cb40-086e-44b8-b08a-2ec9605fc832","Type":"ContainerStarted","Data":"47326be5b50d86e001c5d4adbdaa98297e6e4f2044fb23ab2963d16b5b9794e3"}
Sep 10 16:19:48.723375 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:48.723036    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:48.741846 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:48.741780    2560 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf" podStartSLOduration=2.741759366 podStartE2EDuration="2.741759366s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-10 16:19:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-10 16:19:48.740070069 +0000 UTC m=+2267.120017136" watchObservedRunningTime="2026-09-10 16:19:48.741759366 +0000 UTC m=+2267.121706433"
Sep 10 16:19:49.735403 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:49.735366    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sckvp_7b644548-5bd5-4c2d-bf6d-5eb5230543c3/dns/0.log"
Sep 10 16:19:49.760591 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:49.760563    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-sckvp_7b644548-5bd5-4c2d-bf6d-5eb5230543c3/kube-rbac-proxy/0.log"
Sep 10 16:19:49.789020 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:49.788988    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-8j8vp_54ba45f6-e615-4e8a-8365-51197e8908f6/dns-node-resolver/0.log"
Sep 10 16:19:50.464085 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:50.464040    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-ctnsr_79fa45c1-bfe8-4cfb-b02b-18381665958b/node-ca/0.log"
Sep 10 16:19:51.720780 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:51.720743    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-dmqnm_740e90ae-dfe3-4404-9ab5-789b28590415/kube-auth-proxy/0.log"
Sep 10 16:19:52.877363 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:52.877309    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-frshv_4e8a7cbe-d623-4984-afaa-711821ffcbc6/serve-healthcheck-canary/0.log"
Sep 10 16:19:53.409261 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:53.409208    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2pl8c_c151edf8-7c9c-4008-8149-f18da1a400b1/kube-rbac-proxy/0.log"
Sep 10 16:19:53.432776 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:53.432736    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2pl8c_c151edf8-7c9c-4008-8149-f18da1a400b1/exporter/0.log"
Sep 10 16:19:53.459668 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:53.459637    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-2pl8c_c151edf8-7c9c-4008-8149-f18da1a400b1/extractor/0.log"
Sep 10 16:19:54.737107 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:54.737078    2560 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-2t5gt/perf-node-gather-daemonset-ch9mf"
Sep 10 16:19:56.296967 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:56.296922    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf_1e9c74bf-07f2-429e-9e3e-968e6dffd734/manager/0.log"
Sep 10 16:19:56.334221 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:56.334189    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-8fbcf4bcf-6wlwf_1e9c74bf-07f2-429e-9e3e-968e6dffd734/copy-manifests/0.log"
Sep 10 16:19:57.819714 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:57.819685    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-f8dc47f5d-wqsj4_4508ba2a-caab-4822-b231-80b4211bf721/manager/0.log"
Sep 10 16:19:57.907633 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:57.907569    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-dmqnm_740e90ae-dfe3-4404-9ab5-789b28590415/kube-auth-proxy/0.log"
Sep 10 16:19:59.801437 ip-10-0-138-241 kubenswrapper[2560]: I0910 16:19:59.801402    2560 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6896b79569-dmqnm_740e90ae-dfe3-4404-9ab5-789b28590415/kube-auth-proxy/0.log"