Sep 18 22:14:48.461067 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 18 22:14:48.461096 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 18 22:14:48.461104 ip-10-0-129-37 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 18 22:14:48.461339 ip-10-0-129-37 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 18 22:14:58.843526 ip-10-0-129-37 systemd[1]: kubelet.service: Failed to schedule restart job: Unit openvswitch.service not found.
Sep 18 22:14:58.843540 ip-10-0-129-37 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 7bdd981ffa464eebbccdcfec704d490d --
Sep 18 22:16:47.701312 ip-10-0-129-37 systemd[1]: Starting Kubernetes Kubelet...
Sep 18 22:16:48.104456 ip-10-0-129-37 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 22:16:48.104456 ip-10-0-129-37 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 18 22:16:48.104456 ip-10-0-129-37 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 22:16:48.104456 ip-10-0-129-37 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 18 22:16:48.104456 ip-10-0-129-37 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 18 22:16:48.105101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.104517    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 18 22:16:48.108657 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108635    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.108657 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108654    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.108657 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108660    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.108657 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108663    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108667    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108670    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108673    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108675    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108678    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108681    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108684    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108695    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108701    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108705    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108710    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108714    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108717    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108720    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108723    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108725    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108728    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108730    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.108825 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108734    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108736    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108739    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108745    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108748    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108751    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108754    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108760    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108765    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108768    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108771    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108774    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108776    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108779    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108782    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108785    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108788    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108790    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108793    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108796    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.109290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108798    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108801    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108803    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108806    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108808    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108811    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108813    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108816    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108818    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108821    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108824    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108827    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108829    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108832    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108835    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108837    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108840    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108842    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108845    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108848    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.109769 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108850    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108853    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108855    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108859    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108862    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108864    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108867    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108870    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108873    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108876    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108878    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108881    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108884    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108886    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108889    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108891    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108894    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108897    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108899    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108901    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.110293 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108904    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108907    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108910    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108912    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108915    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108917    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108920    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108922    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108925    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108928    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108930    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108933    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108935    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108938    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108940    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108943    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.108945    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109630    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109637    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.110773 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109641    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109643    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109646    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109649    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109652    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109655    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109657    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109660    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109662    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109665    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109668    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109671    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109673    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109676    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109678    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109682    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109685    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109687    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109690    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109692    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.111254 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109695    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109697    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109700    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109703    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109706    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109709    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109711    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109714    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109716    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109719    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109721    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109724    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109727    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109731    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109733    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109736    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109738    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109741    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109743    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.111808 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109746    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109749    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109751    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109754    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109756    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109759    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109761    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109764    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109766    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109770    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109773    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109776    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109778    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109781    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109784    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109786    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109789    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109792    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109795    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109797    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.112291 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109800    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109802    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109805    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109807    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109810    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109813    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109815    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109818    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109820    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109823    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109825    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109828    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109830    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109834    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109838    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109842    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109845    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109848    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109850    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.112818 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109853    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109855    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109858    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109861    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109864    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109867    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109870    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109873    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109876    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109880    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109883    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109886    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109889    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109892    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109895    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109898    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109901    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109903    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.109906    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110262    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.113294 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110268    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110270    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110273    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110275    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110278    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110281    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110283    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110286    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110289    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110291    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110294    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110297    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110299    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110302    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110305    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110308    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110312    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110316    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110318    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110321    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.113774 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110327    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110330    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110333    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110335    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110338    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110341    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110344    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110346    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110349    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110351    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110354    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110357    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110359    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110362    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110364    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110366    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110369    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110372    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110374    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110376    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.114290 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110379    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110382    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110384    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110387    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110389    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110392    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110395    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110398    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110401    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110403    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110406    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110408    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110412    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110416    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110418    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110421    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110423    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110426    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110428    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.114787 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110431    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110434    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110436    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110439    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110441    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110444    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110446    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110449    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110451    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110454    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110456    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110459    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110461    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110464    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110466    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110468    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110472    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110476    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110479    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.115273 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110482    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110485    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110487    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110491    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110494    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110496    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110500    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110503    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110507    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110509    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110512    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110515    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110517    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110519    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110522    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110525    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110527    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110530    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110534    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.110537    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.115722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110613    2551 flags.go:64] FLAG: --address="0.0.0.0"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110655    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110662    2551 flags.go:64] FLAG: --anonymous-auth="true"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110666    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110670    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110673    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110678    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110681    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110685    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110688    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110691    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110695    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110698    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110701    2551 flags.go:64] FLAG: --cgroup-root=""
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110703    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110706    2551 flags.go:64] FLAG: --client-ca-file=""
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110710    2551 flags.go:64] FLAG: --cloud-provider="external"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110713    2551 flags.go:64] FLAG: --cluster-dns="[]"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110719    2551 flags.go:64] FLAG: --cluster-domain=""
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110722    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110726    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110729    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110732    2551 flags.go:64] FLAG: --container-log-max-files="5"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110736    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 18 22:16:48.116212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110739    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110742    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110745    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110748    2551 flags.go:64] FLAG: --contention-profiling="false"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110751    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110754    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110757    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110759    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110763    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110766    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110769    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110772    2551 flags.go:64] FLAG: --enable-load-reader="false"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110775    2551 flags.go:64] FLAG: --enable-server="true"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110778    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110782    2551 flags.go:64] FLAG: --event-burst="100"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110785    2551 flags.go:64] FLAG: --event-qps="50"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110788    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110791    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110794    2551 flags.go:64] FLAG: --eviction-hard=""
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110797    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110800    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110803    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110806    2551 flags.go:64] FLAG: --eviction-soft=""
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110808    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110811    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 18 22:16:48.116791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110814    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110819    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110822    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110825    2551 flags.go:64] FLAG: --fail-swap-on="true"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110828    2551 flags.go:64] FLAG: --feature-gates=""
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110831    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110834    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110837    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110840    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110843    2551 flags.go:64] FLAG: --healthz-port="10248"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110846    2551 flags.go:64] FLAG: --help="false"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110849    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110852    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110855    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110858    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110861    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110865    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110867    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110870    2551 flags.go:64] FLAG: --image-service-endpoint=""
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110873    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110876    2551 flags.go:64] FLAG: --kube-api-burst="100"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110878    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110881    2551 flags.go:64] FLAG: --kube-api-qps="50"
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110884    2551 flags.go:64] FLAG: --kube-reserved=""
Sep 18 22:16:48.117471 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110887    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110890    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110892    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110895    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110898    2551 flags.go:64] FLAG: --lock-file=""
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110901    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110903    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110906    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110911    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110916    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110920    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110923    2551 flags.go:64] FLAG: --logging-format="text"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110926    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110929    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110932    2551 flags.go:64] FLAG: --manifest-url=""
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110935    2551 flags.go:64] FLAG: --manifest-url-header=""
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110939    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110942    2551 flags.go:64] FLAG: --max-open-files="1000000"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110946    2551 flags.go:64] FLAG: --max-pods="110"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110949    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110952    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110954    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110957    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110960    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110963    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 18 22:16:48.118096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110966    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110973    2551 flags.go:64] FLAG: --node-status-max-images="50"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110976    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110979    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110982    2551 flags.go:64] FLAG: --pod-cidr=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110984    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:a6abe77af61c5f2e3985a83e7d43538aefc38b1b0444127da14ee0e8cf8f76fc"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110989    2551 flags.go:64] FLAG: --pod-manifest-path=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110992    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110995    2551 flags.go:64] FLAG: --pods-per-core="0"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.110998    2551 flags.go:64] FLAG: --port="10250"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111001    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111004    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06eac63fe84890d47"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111007    2551 flags.go:64] FLAG: --qos-reserved=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111010    2551 flags.go:64] FLAG: --read-only-port="10255"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111012    2551 flags.go:64] FLAG: --register-node="true"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111015    2551 flags.go:64] FLAG: --register-with-taints=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111020    2551 flags.go:64] FLAG: --registry-burst="10"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111041    2551 flags.go:64] FLAG: --registry-qps="5"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111046    2551 flags.go:64] FLAG: --reserved-cpus=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111049    2551 flags.go:64] FLAG: --reserved-memory=""
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111057    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111060    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111063    2551 flags.go:64] FLAG: --rotate-certificates="false"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111066    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111068    2551 flags.go:64] FLAG: --runonce="false"
Sep 18 22:16:48.118704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111071    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111074    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111077    2551 flags.go:64] FLAG: --seccomp-default="false"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111080    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111083    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111086    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111089    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111092    2551 flags.go:64] FLAG: --storage-driver-password="root"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111094    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111097    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111100    2551 flags.go:64] FLAG: --storage-driver-user="root"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111103    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111106    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111109    2551 flags.go:64] FLAG: --system-cgroups=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111111    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111116    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111119    2551 flags.go:64] FLAG: --tls-cert-file=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111122    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111126    2551 flags.go:64] FLAG: --tls-min-version=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111129    2551 flags.go:64] FLAG: --tls-private-key-file=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111132    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111135    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111138    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111142    2551 flags.go:64] FLAG: --v="2"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111146    2551 flags.go:64] FLAG: --version="false"
Sep 18 22:16:48.119328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111152    2551 flags.go:64] FLAG: --vmodule=""
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111155    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.111158    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111279    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111283    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111287    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111290    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111293    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111296    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111299    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111302    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111304    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111307    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111310    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111312    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111315    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111318    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111320    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111323    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111327    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.119921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111331    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111334    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111337    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111340    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111343    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111348    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111353    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111357    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111361    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111364    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111368    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111371    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111375    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111379    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111382    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111384    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111387    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111389    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111392    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.120475 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111395    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111398    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111400    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111403    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111406    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111408    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111411    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111413    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111417    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111421    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111425    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111428    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111430    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111433    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111436    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111438    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111441    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111443    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111446    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111448    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.120934 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111451    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111453    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111456    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111460    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111462    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111466    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111469    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111472    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111474    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111477    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111479    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111482    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111485    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111487    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111490    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111492    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111495    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111498    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111501    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111503    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.121424 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111506    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111508    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111511    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111514    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111516    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111519    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111521    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111524    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111527    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111529    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111532    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111535    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111537    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111540    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111542    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111546    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111548    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111552    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111555    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111558    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.121941 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111561    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111563    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.111566    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.112141    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.118918    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.118932    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.120541    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.120549    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120610    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120615    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120619    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120622    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120625    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120627    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120630    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.122443 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120633    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120636    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120638    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120641    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120643    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120646    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120649    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120652    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120654    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120657    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120659    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120662    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120665    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120667    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120670    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120673    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120675    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120678    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120681    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120683    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.122816 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120686    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120699    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120705    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120708    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120712    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120716    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120719    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120722    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120725    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120727    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120730    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120733    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120736    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120738    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120741    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120744    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120746    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120749    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120751    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.123303 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120754    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120756    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120759    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120762    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120764    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120767    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120769    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120772    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120775    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120777    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120780    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120782    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120785    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120788    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120791    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120794    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120797    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120800    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120802    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120805    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.123794 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120807    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120810    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120813    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120815    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120818    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120820    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120823    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120826    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120828    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120831    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120833    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120836    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120840    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120843    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120846    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120848    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120851    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120854    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120857    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.124309 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120859    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120862    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120864    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120867    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120870    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120872    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120875    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120878    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120881    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120884    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120893    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120896    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120899    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.120901    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.124791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.120906    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121049    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121056    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121059    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121062    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121066    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121068    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121072    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121074    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121077    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121079    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121082    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121084    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121087    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121089    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121092    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121095    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121097    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121100    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121102    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 18 22:16:48.125209 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121105    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121107    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121110    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121112    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121115    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121119    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121121    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121124    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121127    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121136    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121139    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121142    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121144    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121147    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121150    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121152    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121155    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121158    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121160    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121163    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 18 22:16:48.125707 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121165    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121168    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121170    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121173    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121175    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121178    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121180    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121183    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121187    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121190    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121193    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121196    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121198    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121201    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121203    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121206    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121208    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121211    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121214    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 18 22:16:48.126200 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121216    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121219    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121223    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121234    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121237    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121239    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121242    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121245    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121248    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121250    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121253    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121256    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121258    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121261    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121273    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121276    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121278    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121281    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121284    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121286    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 18 22:16:48.126652 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121289    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121292    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121294    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121297    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121300    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121302    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121305    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121307    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121310    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121312    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121315    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121317    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121320    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121324    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121327    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121330    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121339    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121341    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121344    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 18 22:16:48.127157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121347    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:48.121350    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.121355    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.121461    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.123592    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.124661    2551 server.go:1019] "Starting client certificate rotation"
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.124756    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 22:16:48.127635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.124790    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 18 22:16:48.128263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.128249    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 22:16:48.130179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.130138    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 18 22:16:48.139670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.139654    2551 log.go:25] "Validated CRI v1 runtime API"
Sep 18 22:16:48.141149 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.141135    2551 log.go:25] "Validated CRI v1 image API"
Sep 18 22:16:48.143514 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.143498    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 18 22:16:48.149128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.149108    2551 fs.go:135] Filesystem UUIDs: map[19765ec4-b3fd-4d46-9ec6-b892a1c16ab2:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2 ace85174-6539-4d30-a55b-0d2aef8614dd:/dev/nvme0n1p3]
Sep 18 22:16:48.149184 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.149124    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 18 22:16:48.155689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.155577    2551 manager.go:216] Machine: {Timestamp:2026-09-18 22:16:48.154192536 +0000 UTC m=+0.335777077 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3099700 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2f64ad74c1fb04142ac12be6a46ea0 SystemUUID:ec2f64ad-74c1-fb04-142a-c12be6a46ea0 BootID:7bdd981f-fa46-4eeb-bccd-cfec704d490d Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:f0:6d:79:e7 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:f0:6d:79:e7 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:16:37:9f:b0:d5:d2 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 18 22:16:48.155689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.155682    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 18 22:16:48.155801 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.155752    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.141.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260914-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 18 22:16:48.158087 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.158060    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 18 22:16:48.158211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.158090    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-129-37.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 18 22:16:48.159084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.159075    2551 topology_manager.go:138] "Creating topology manager with none policy"
Sep 18 22:16:48.159120 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.159087    2551 container_manager_linux.go:309] "Creating device plugin manager"
Sep 18 22:16:48.159120 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.159099    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 22:16:48.159898 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.159887    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 18 22:16:48.159962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.159954    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 18 22:16:48.161151 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.161132    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 22:16:48.163575 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163563    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 22:16:48.163687 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163677    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 18 22:16:48.163722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163708    2551 kubelet.go:486] "Attempting to sync node with API server"
Sep 18 22:16:48.163722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163718    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 18 22:16:48.163782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163739    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 18 22:16:48.163782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163752    2551 kubelet.go:398] "Adding apiserver pod source"
Sep 18 22:16:48.163782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.163771    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 18 22:16:48.164132 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.164120    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 22:16:48.164172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.164155    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 18 22:16:48.166761 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.166739    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.13-6.rhaos4.21.git6253162.el9" apiVersion="v1"
Sep 18 22:16:48.169003 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.168989    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 18 22:16:48.169075 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.169019    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 18 22:16:48.170472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170457    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170479    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170485    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170492    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170500    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170505    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170518    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170523    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 18 22:16:48.170527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170529    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 18 22:16:48.170734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170535    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 18 22:16:48.170734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170543    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 18 22:16:48.170734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.170548    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 18 22:16:48.171445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.171433    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 18 22:16:48.171445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.171445    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 18 22:16:48.176352 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.176331    2551 server.go:1268] "Started kubelet"
Sep 18 22:16:48.176454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.176408    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 18 22:16:48.176508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.176443    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 18 22:16:48.176554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.176526    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 18 22:16:48.176735 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.176714    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-129-37.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 18 22:16:48.176799 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.176770    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-129-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 18 22:16:48.176859 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.176847    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 18 22:16:48.177321 ip-10-0-129-37 systemd[1]: Started Kubernetes Kubelet.
Sep 18 22:16:48.177634 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.177618    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 18 22:16:48.181610 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.181557    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 18 22:16:48.181686 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.181674    2551 server.go:310] "Adding debug handlers to kubelet server"
Sep 18 22:16:48.184298 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.183324    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d68a286e114735  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-18 22:16:48.176301877 +0000 UTC m=+0.357886414,LastTimestamp:2026-09-18 22:16:48.176301877 +0000 UTC m=+0.357886414,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 18 22:16:48.184449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.184438    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 18 22:16:48.184502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.184448    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 18 22:16:48.184847 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.184505    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.185043 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185015    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 18 22:16:48.185043 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185043    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 18 22:16:48.185164 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185124    2551 reconstruct.go:97] "Volume reconstruction finished"
Sep 18 22:16:48.185164 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185131    2551 reconciler.go:29] "Reconciler: start to sync state"
Sep 18 22:16:48.185164 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185138    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 18 22:16:48.185312 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185248    2551 factory.go:55] Registering systemd factory
Sep 18 22:16:48.185312 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185264    2551 factory.go:223] Registration of the systemd container factory successfully
Sep 18 22:16:48.185487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185471    2551 factory.go:153] Registering CRI-O factory
Sep 18 22:16:48.185487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185486    2551 factory.go:223] Registration of the crio container factory successfully
Sep 18 22:16:48.185612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185578    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 18 22:16:48.185612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185596    2551 factory.go:103] Registering Raw factory
Sep 18 22:16:48.185612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.185612    2551 manager.go:1196] Started watching for new ooms in manager
Sep 18 22:16:48.186042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.186014    2551 manager.go:349] Starting recovery of all containers
Sep 18 22:16:48.190320 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.190289    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 18 22:16:48.190444 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.190405    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-129-37.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 18 22:16:48.199431 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.199410    2551 manager.go:354] Recovery completed
Sep 18 22:16:48.205193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.205181    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.206797 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.206782    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.206843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.206813    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.206843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.206828    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.207337 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207324    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.207384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207326    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 18 22:16:48.207384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207355    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 18 22:16:48.207462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207406    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 18 22:16:48.207930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207918    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.207961 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207944    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.207961 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.207956    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.209156 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.209082    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d68a286fe29c76  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-18 22:16:48.206797942 +0000 UTC m=+0.388382486,LastTimestamp:2026-09-18 22:16:48.206797942 +0000 UTC m=+0.388382486,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 18 22:16:48.209401 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.209389    2551 policy_none.go:49] "None policy: Start"
Sep 18 22:16:48.209439 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.209405    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 18 22:16:48.209439 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.209415    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 18 22:16:48.211067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.211055    2551 policy_none.go:47] "Start"
Sep 18 22:16:48.216108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.216090    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwzpj"
Sep 18 22:16:48.217800 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.217740    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d68a286fe2f596  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-129-37.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-18 22:16:48.206820758 +0000 UTC m=+0.388405296,LastTimestamp:2026-09-18 22:16:48.206820758 +0000 UTC m=+0.388405296,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 18 22:16:48.220289 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.220222    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d68a286fe329c8  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientPID,Message:Node ip-10-0-129-37.ec2.internal status is now: NodeHasSufficientPID,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-18 22:16:48.20683412 +0000 UTC m=+0.388418661,LastTimestamp:2026-09-18 22:16:48.20683412 +0000 UTC m=+0.388418661,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 18 22:16:48.224581 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.224555    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-mwzpj"
Sep 18 22:16:48.228277 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.228220    2551 event.go:359] "Server rejected event (will not retry!)" err="events \"ip-10-0-129-37.ec2.internal.18d68a286fe29c76\" is forbidden: User \"system:anonymous\" cannot patch resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-129-37.ec2.internal.18d68a286fe29c76  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-129-37.ec2.internal,UID:ip-10-0-129-37.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-129-37.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-129-37.ec2.internal,},FirstTimestamp:2026-09-18 22:16:48.206797942 +0000 UTC m=+0.388382486,LastTimestamp:2026-09-18 22:16:48.207929978 +0000 UTC m=+0.389514515,Count:2,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-129-37.ec2.internal,}"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.237762    2551 manager.go:337] "Starting Device Plugin manager"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.237790    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.237801    2551 server.go:87] "Starting device plugin registration server"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.238067    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.238077    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.238160    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.238259    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.238267    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.238730    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 18 22:16:48.264532 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.238777    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.282982 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.282961    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 18 22:16:48.284038 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.284013    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 18 22:16:48.284124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.284068    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 18 22:16:48.284124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.284090    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 18 22:16:48.284216 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.284129    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 18 22:16:48.286274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.286258    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:48.339152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.339131    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.341465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.341445    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.341553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.341473    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.341553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.341483    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.341553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.341511    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.348511 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.348496    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.348571 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.348516    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-129-37.ec2.internal\": node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.383102 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.383046    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.385088 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.385075    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"]
Sep 18 22:16:48.385144 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.385126    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.385529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.385506    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.385582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.385536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.386455 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.386433    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.386526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.386464    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.386526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.386476    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.388594 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.388582    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.388709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.388696    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.388752 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.388722    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.389969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.389945    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.389969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.389970    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.390101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.389982    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.390101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.389987    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.390101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.390007    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.390101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.390020    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.392563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.392540    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.392630 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.392570    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 18 22:16:48.393213 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.393197    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientMemory"
Sep 18 22:16:48.393285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.393224    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasNoDiskPressure"
Sep 18 22:16:48.393285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.393233    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeHasSufficientPID"
Sep 18 22:16:48.426523 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.426502    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-37.ec2.internal\" not found" node="ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.430553 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.430539    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-129-37.ec2.internal\" not found" node="ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.483359 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.483342    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.486607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.486592    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.486654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.486616    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d86a8c0ee26cd8a3423d7b77f0c8af-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"b2d86a8c0ee26cd8a3423d7b77f0c8af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.486654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.486633    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.486716 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.486697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.486716 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.486700    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/2d36f920ee1c195e31743d5172b70185-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal\" (UID: \"2d36f920ee1c195e31743d5172b70185\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.584388 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.584370    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.587614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.587600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d86a8c0ee26cd8a3423d7b77f0c8af-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"b2d86a8c0ee26cd8a3423d7b77f0c8af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.587663 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.587632    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/b2d86a8c0ee26cd8a3423d7b77f0c8af-config\") pod \"kube-apiserver-proxy-ip-10-0-129-37.ec2.internal\" (UID: \"b2d86a8c0ee26cd8a3423d7b77f0c8af\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.685047 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.685000    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.730378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.730358    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.734463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:48.734444    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:48.785571 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.785549    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.886004 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.885969    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:48.986436 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:48.986390    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:49.086947 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.086927    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-129-37.ec2.internal\" not found"
Sep 18 22:16:49.095642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.095627    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:49.124363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.124345    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 18 22:16:49.124700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.124457    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 22:16:49.124700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.124486    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 18 22:16:49.163940 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.163919    2551 apiserver.go:52] "Watching apiserver"
Sep 18 22:16:49.165296 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.165278    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:49.170196 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.170176    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:49.174373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.174358    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 18 22:16:49.174706 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.174687    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/global-pull-secret-syncer-w6mzb","openshift-dns/node-resolver-v7kdp","openshift-image-registry/node-ca-mbrwj","openshift-multus/multus-additional-cni-plugins-sd9wb","openshift-multus/network-metrics-daemon-ldvsn","openshift-network-diagnostics/network-check-target-z8hff","openshift-ovn-kubernetes/ovnkube-node-g7g7q","kube-system/konnectivity-agent-pjg4b","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js","openshift-cluster-node-tuning-operator/tuned-jskkx","openshift-multus/multus-psjql","openshift-network-operator/iptables-alerter-mlkh8"]
Sep 18 22:16:49.177474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.177454    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.180024 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.180005    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.182181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.182163    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.182266 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.182222    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:49.184578 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.184550    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:49.184578 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.184569    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 18 22:16:49.184951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.184933    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.188098 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.188010    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.188399 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.188380    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.188538 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.188518    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.188612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.188593    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-f9cpv\""
Sep 18 22:16:49.188673 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.188627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-zwjfq\""
Sep 18 22:16:49.190310 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190294    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0777467-72d9-4cb1-89de-35f82dc26ec2-serviceca\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.190402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190320    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.190402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190339    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.190402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zfnxk\" (UniqueName: \"kubernetes.io/projected/ff71cfa3-3462-41c0-93fe-6118322980e0-kube-api-access-zfnxk\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.190516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-run\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190419    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-system-cni-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.190516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190433    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-host\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff71cfa3-3462-41c0-93fe-6118322980e0-tmp-dir\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190537    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-modprobe-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190552    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-conf\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190571    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-sys\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190595    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-var-lib-kubelet\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190610    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysconfig\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190639    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-tmp\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190662    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mcjgr\" (UniqueName: \"kubernetes.io/projected/c1d01c54-9e81-4b86-b117-fa856a0512e9-kube-api-access-mcjgr\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.190709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190694    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-dbus\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190724    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kqxrj\" (UniqueName: \"kubernetes.io/projected/27cfab9a-6928-47c8-a680-cfbbd86cd311-kube-api-access-kqxrj\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190751    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff71cfa3-3462-41c0-93fe-6118322980e0-hosts-file\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-kubernetes\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-kubelet-config\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9wg6b\" (UniqueName: \"kubernetes.io/projected/c0777467-72d9-4cb1-89de-35f82dc26ec2-kube-api-access-9wg6b\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190849    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-lib-modules\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190870    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-cnibin\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-systemd\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-tuned\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.191167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.190958    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0777467-72d9-4cb1-89de-35f82dc26ec2-host\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.191649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.191185    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.191649 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.191248    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:16:49.193662 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.193642    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:49.193745 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.193696    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:49.193865 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.193846    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.193865 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.193857    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.193974 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.193859    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 18 22:16:49.194185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194168    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.194235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194200    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 18 22:16:49.194235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194204    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 18 22:16:49.194315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194250    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.194315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194210    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-4rksw\""
Sep 18 22:16:49.194477 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194461    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-4p6sp\""
Sep 18 22:16:49.194543 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.194528    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 18 22:16:49.196664 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.196639    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.196914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.196901    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.198167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.198148    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.199486 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.199469    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.202030 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.202012    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.203589 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.203567    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 18 22:16:49.203589 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.203578    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 18 22:16:49.204664 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.204647    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.207268 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207243    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.207415 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207391    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 18 22:16:49.207488 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207416    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 18 22:16:49.207488 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207459    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.207598 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207488    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 18 22:16:49.207656 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.207609    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.208847    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-dvgzf\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209324    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-jg4r5\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209369    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-95qgb\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209596    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-cbrk8\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209661    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209833    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.209954    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.210072    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 18 22:16:49.211097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.210379    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 18 22:16:49.214503 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.214475    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 18 22:16:49.214600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.214549    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 18 22:16:49.214835 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.214821    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-867hh\""
Sep 18 22:16:49.215351 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.215333    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 18 22:16:49.218987 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.218963    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 18 22:16:49.225119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.225094    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal"]
Sep 18 22:16:49.226070 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.226052    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 22:16:49.226168 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.226152    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"
Sep 18 22:16:49.230397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.230356    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-17 22:11:48 +0000 UTC" deadline="2028-03-17 18:53:20.122666196 +0000 UTC"
Sep 18 22:16:49.230397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.230394    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13100h36m30.892274808s"
Sep 18 22:16:49.247805 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.247789    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 18 22:16:49.247994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.247979    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal"]
Sep 18 22:16:49.258905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.258882    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-wh7bg"
Sep 18 22:16:49.269545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.269527    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-wh7bg"
Sep 18 22:16:49.285976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.285959    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 18 22:16:49.291751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291732    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-cnibin\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.291847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-tuned\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.291847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.291847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291798    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0777467-72d9-4cb1-89de-35f82dc26ec2-host\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.291847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291818    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-config\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291844    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-cnibin\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291845    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291878    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c0777467-72d9-4cb1-89de-35f82dc26ec2-host\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291889    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-run\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291912    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-systemd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.291927    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291979    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-run\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.291930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30939455-de05-41c9-8edb-a4b157b28534-agent-certs\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.292001    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:16:49.7919811 +0000 UTC m=+1.973565639 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292040    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-cni-binary-copy\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.292094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292095    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-etc-kubernetes\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292121    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx76z\" (UniqueName: \"kubernetes.io/projected/a1b653c8-c7da-477e-8c16-5885f92b483c-kube-api-access-cx76z\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-host\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292170    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292202    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-host\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292194    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpzm9\" (UniqueName: \"kubernetes.io/projected/db877126-ccbf-4727-a199-5bd4527f08c4-kube-api-access-tpzm9\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292245    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-bin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292274    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-hostroot\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-sys\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-var-lib-kubelet\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-log-socket\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-sys\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-registration-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-var-lib-kubelet\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-k8s-cni-cncf-io\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292450    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysconfig\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.292647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292493    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-netns\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30939455-de05-41c9-8edb-a4b157b28534-konnectivity-ca\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292529    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysconfig\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292544    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kr9l4\" (UniqueName: \"kubernetes.io/projected/69d0f323-1928-4b8a-b248-e04808543b76-kube-api-access-kr9l4\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292589    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kqxrj\" (UniqueName: \"kubernetes.io/projected/27cfab9a-6928-47c8-a680-cfbbd86cd311-kube-api-access-kqxrj\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292617    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292647    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff71cfa3-3462-41c0-93fe-6118322980e0-hosts-file\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292659    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292693    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-binary-copy\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292670    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-kubernetes\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292725    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-kubernetes\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292755    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-kubelet-config\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292782    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-etc-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ff71cfa3-3462-41c0-93fe-6118322980e0-hosts-file\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-kubelet-config\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292819    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-system-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292849    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-socket-dir-parent\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.293490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1b653c8-c7da-477e-8c16-5885f92b483c-iptables-alerter-script\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292900    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-systemd\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292918    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0777467-72d9-4cb1-89de-35f82dc26ec2-serviceca\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292980    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nht92\" (UniqueName: \"kubernetes.io/projected/e440d077-757a-4d92-a61e-357569199de5-kube-api-access-nht92\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.292995    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-systemd\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293062    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-kubelet\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293087    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-systemd-units\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293110    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1b653c8-c7da-477e-8c16-5885f92b483c-host-slash\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293172    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zfnxk\" (UniqueName: \"kubernetes.io/projected/ff71cfa3-3462-41c0-93fe-6118322980e0-kube-api-access-zfnxk\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293208    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-sys-fs\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293266    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-cnibin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-system-cni-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-system-cni-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-node-log\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-netd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.294251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293423    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/c0777467-72d9-4cb1-89de-35f82dc26ec2-serviceca\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293428    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-device-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293481    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-netns\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293508    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-multus\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293546    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293562    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293585    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff71cfa3-3462-41c0-93fe-6118322980e0-tmp-dir\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293627    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-modprobe-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293688    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-conf\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293701    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/27cfab9a-6928-47c8-a680-cfbbd86cd311-tuning-conf-dir\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293717    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-bin\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293736    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-modprobe-d\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293755    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovn-node-metrics-cert\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293787    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-tmp\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293822    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-sysctl-conf\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293834    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mcjgr\" (UniqueName: \"kubernetes.io/projected/c1d01c54-9e81-4b86-b117-fa856a0512e9-kube-api-access-mcjgr\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293866    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-dbus\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.295099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293883    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-slash\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293918    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-socket-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293937    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ff71cfa3-3462-41c0-93fe-6118322980e0-tmp-dir\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293958    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-conf-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293984    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-multus-daemon-config\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.293996    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-dbus\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294025    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294051    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-ovn\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294108    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-script-lib\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294131    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c6rk7\" (UniqueName: \"kubernetes.io/projected/fcd4e641-bee5-4d87-b8bb-80e070cf7461-kube-api-access-c6rk7\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294158    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-multus-certs\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294198    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9wg6b\" (UniqueName: \"kubernetes.io/projected/c0777467-72d9-4cb1-89de-35f82dc26ec2-kube-api-access-9wg6b\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294259    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-var-lib-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294286    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294311    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294347    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294392    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-lib-modules\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294416    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-env-overrides\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294441    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-kubelet\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/c1d01c54-9e81-4b86-b117-fa856a0512e9-lib-modules\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.294690    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/27cfab9a-6928-47c8-a680-cfbbd86cd311-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.296377    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-tmp\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.296669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.296497    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/c1d01c54-9e81-4b86-b117-fa856a0512e9-etc-tuned\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.303361 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.303257    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9wg6b\" (UniqueName: \"kubernetes.io/projected/c0777467-72d9-4cb1-89de-35f82dc26ec2-kube-api-access-9wg6b\") pod \"node-ca-mbrwj\" (UID: \"c0777467-72d9-4cb1-89de-35f82dc26ec2\") " pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.303361 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.303258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zfnxk\" (UniqueName: \"kubernetes.io/projected/ff71cfa3-3462-41c0-93fe-6118322980e0-kube-api-access-zfnxk\") pod \"node-resolver-v7kdp\" (UID: \"ff71cfa3-3462-41c0-93fe-6118322980e0\") " pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.303918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.303881    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mcjgr\" (UniqueName: \"kubernetes.io/projected/c1d01c54-9e81-4b86-b117-fa856a0512e9-kube-api-access-mcjgr\") pod \"tuned-jskkx\" (UID: \"c1d01c54-9e81-4b86-b117-fa856a0512e9\") " pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.304001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.303892    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kqxrj\" (UniqueName: \"kubernetes.io/projected/27cfab9a-6928-47c8-a680-cfbbd86cd311-kube-api-access-kqxrj\") pod \"multus-additional-cni-plugins-sd9wb\" (UID: \"27cfab9a-6928-47c8-a680-cfbbd86cd311\") " pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.363013 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.362980    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb2d86a8c0ee26cd8a3423d7b77f0c8af.slice/crio-5f960f0939a57ae02f0dbd8a06093aebb1670ed9f03baa8831d321a7b21ea85b WatchSource:0}: Error finding container 5f960f0939a57ae02f0dbd8a06093aebb1670ed9f03baa8831d321a7b21ea85b: Status 404 returned error can't find the container with id 5f960f0939a57ae02f0dbd8a06093aebb1670ed9f03baa8831d321a7b21ea85b
Sep 18 22:16:49.363497 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.363480    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d36f920ee1c195e31743d5172b70185.slice/crio-36859d3767ff86d94b6d32a05cbad0d082680922d29a0ad8ab9b2fec676702b2 WatchSource:0}: Error finding container 36859d3767ff86d94b6d32a05cbad0d082680922d29a0ad8ab9b2fec676702b2: Status 404 returned error can't find the container with id 36859d3767ff86d94b6d32a05cbad0d082680922d29a0ad8ab9b2fec676702b2
Sep 18 22:16:49.371642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.369474    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:16:49.395284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395260    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-log-socket\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395286    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-registration-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-k8s-cni-cncf-io\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395323    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-netns\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395348    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30939455-de05-41c9-8edb-a4b157b28534-konnectivity-ca\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395373    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kr9l4\" (UniqueName: \"kubernetes.io/projected/69d0f323-1928-4b8a-b248-e04808543b76-kube-api-access-kr9l4\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395378    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-registration-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395390    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-log-socket\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395401    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-etc-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395407    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-k8s-cni-cncf-io\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-system-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395412    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-netns\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395466    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-etc-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395484    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-system-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395498    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-socket-dir-parent\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1b653c8-c7da-477e-8c16-5885f92b483c-iptables-alerter-script\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395560    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nht92\" (UniqueName: \"kubernetes.io/projected/e440d077-757a-4d92-a61e-357569199de5-kube-api-access-nht92\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395597    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-socket-dir-parent\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395605    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-kubelet\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.395654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-systemd-units\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395677    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1b653c8-c7da-477e-8c16-5885f92b483c-host-slash\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395690    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-systemd-units\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395640    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-kubelet\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395711    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395743    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/a1b653c8-c7da-477e-8c16-5885f92b483c-host-slash\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-sys-fs\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395783    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-sys-fs\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395807    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-cnibin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395830    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-node-log\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-netd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395860    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-device-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395882    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-netns\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/30939455-de05-41c9-8edb-a4b157b28534-konnectivity-ca\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395908    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-multus\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-netd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.396172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395936    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-bin\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395937    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-device-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395964    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-netns\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-node-log\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395975    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-multus\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.395997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-cni-bin\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396002    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-cnibin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396004    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovn-node-metrics-cert\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/a1b653c8-c7da-477e-8c16-5885f92b483c-iptables-alerter-script\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396092    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-slash\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-slash\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396144    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-socket-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396160    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-conf-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396175    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-multus-daemon-config\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396205    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-ovn\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396243    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-script-lib\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.397034 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396268    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396277    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c6rk7\" (UniqueName: \"kubernetes.io/projected/fcd4e641-bee5-4d87-b8bb-80e070cf7461-kube-api-access-c6rk7\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396268    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-conf-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396283    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-socket-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-ovn\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396282    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-host-run-ovn-kubernetes\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-multus-certs\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396346    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-run-multus-certs\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396379    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-var-lib-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396420    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-var-lib-openvswitch\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396458    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396470    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-kubelet-dir\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396485    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-env-overrides\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396561    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/db877126-ccbf-4727-a199-5bd4527f08c4-etc-selinux\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.398067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396562    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-kubelet\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396628    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-config\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396648    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-systemd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396667    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-multus-cni-dir\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396672    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30939455-de05-41c9-8edb-a4b157b28534-agent-certs\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396705    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-cni-binary-copy\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-kubelet\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396731    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-etc-kubernetes\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396758    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-multus-daemon-config\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-etc-kubernetes\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396835    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx76z\" (UniqueName: \"kubernetes.io/projected/a1b653c8-c7da-477e-8c16-5885f92b483c-kube-api-access-cx76z\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396910    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/fcd4e641-bee5-4d87-b8bb-80e070cf7461-run-systemd\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396959    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396984    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-env-overrides\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396989    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpzm9\" (UniqueName: \"kubernetes.io/projected/db877126-ccbf-4727-a199-5bd4527f08c4-kube-api-access-tpzm9\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397061    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-bin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.397081    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:49.399016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397090    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-hostroot\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.397150    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:49.897132839 +0000 UTC m=+2.078717382 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397156    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-hostroot\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.396986    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-script-lib\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397211    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/69d0f323-1928-4b8a-b248-e04808543b76-host-var-lib-cni-bin\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397300    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/69d0f323-1928-4b8a-b248-e04808543b76-cni-binary-copy\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.397573    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovnkube-config\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.398285    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/fcd4e641-bee5-4d87-b8bb-80e070cf7461-ovn-node-metrics-cert\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.399547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.399315    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/30939455-de05-41c9-8edb-a4b157b28534-agent-certs\") pod \"konnectivity-agent-pjg4b\" (UID: \"30939455-de05-41c9-8edb-a4b157b28534\") " pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.405521 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.405495    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:16:49.405521 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.405514    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:16:49.405521 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.405523    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:49.405683 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.405575    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:49.905558248 +0000 UTC m=+2.087142785 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:49.407718 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.407698    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kr9l4\" (UniqueName: \"kubernetes.io/projected/69d0f323-1928-4b8a-b248-e04808543b76-kube-api-access-kr9l4\") pod \"multus-psjql\" (UID: \"69d0f323-1928-4b8a-b248-e04808543b76\") " pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.408058 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.408031    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx76z\" (UniqueName: \"kubernetes.io/projected/a1b653c8-c7da-477e-8c16-5885f92b483c-kube-api-access-cx76z\") pod \"iptables-alerter-mlkh8\" (UID: \"a1b653c8-c7da-477e-8c16-5885f92b483c\") " pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.408424 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.408406    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nht92\" (UniqueName: \"kubernetes.io/projected/e440d077-757a-4d92-a61e-357569199de5-kube-api-access-nht92\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.408815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.408793    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpzm9\" (UniqueName: \"kubernetes.io/projected/db877126-ccbf-4727-a199-5bd4527f08c4-kube-api-access-tpzm9\") pod \"aws-ebs-csi-driver-node-sj2js\" (UID: \"db877126-ccbf-4727-a199-5bd4527f08c4\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.409325 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.409306    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c6rk7\" (UniqueName: \"kubernetes.io/projected/fcd4e641-bee5-4d87-b8bb-80e070cf7461-kube-api-access-c6rk7\") pod \"ovnkube-node-g7g7q\" (UID: \"fcd4e641-bee5-4d87-b8bb-80e070cf7461\") " pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.503829 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.503779    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-v7kdp"
Sep 18 22:16:49.513511 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.513491    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podff71cfa3_3462_41c0_93fe_6118322980e0.slice/crio-82e5488029192e3a7b51497b8170b63cb69e63ad928391335fd3ffcf52dbf54a WatchSource:0}: Error finding container 82e5488029192e3a7b51497b8170b63cb69e63ad928391335fd3ffcf52dbf54a: Status 404 returned error can't find the container with id 82e5488029192e3a7b51497b8170b63cb69e63ad928391335fd3ffcf52dbf54a
Sep 18 22:16:49.518653 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.518639    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-jskkx"
Sep 18 22:16:49.526553 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.526532    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d01c54_9e81_4b86_b117_fa856a0512e9.slice/crio-89f79e234b0599d36a0c03261b6ab6876048e1048fb42fbb9e4e30576c60438c WatchSource:0}: Error finding container 89f79e234b0599d36a0c03261b6ab6876048e1048fb42fbb9e4e30576c60438c: Status 404 returned error can't find the container with id 89f79e234b0599d36a0c03261b6ab6876048e1048fb42fbb9e4e30576c60438c
Sep 18 22:16:49.535912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.535896    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-mbrwj"
Sep 18 22:16:49.540914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.540898    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-sd9wb"
Sep 18 22:16:49.544227 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.544208    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0777467_72d9_4cb1_89de_35f82dc26ec2.slice/crio-5d55cd36fee548d69b13c81cc8744478f6415c5ee6a6a6256dc227b79f98138f WatchSource:0}: Error finding container 5d55cd36fee548d69b13c81cc8744478f6415c5ee6a6a6256dc227b79f98138f: Status 404 returned error can't find the container with id 5d55cd36fee548d69b13c81cc8744478f6415c5ee6a6a6256dc227b79f98138f
Sep 18 22:16:49.549470 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.549450    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod27cfab9a_6928_47c8_a680_cfbbd86cd311.slice/crio-ed407ed204fd1687d54ff363fb647ee2203512bd0e8a77855cf5e7e030924874 WatchSource:0}: Error finding container ed407ed204fd1687d54ff363fb647ee2203512bd0e8a77855cf5e7e030924874: Status 404 returned error can't find the container with id ed407ed204fd1687d54ff363fb647ee2203512bd0e8a77855cf5e7e030924874
Sep 18 22:16:49.552534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.552506    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:16:49.561419 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.561399    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd4e641_bee5_4d87_b8bb_80e070cf7461.slice/crio-14003241d7d50bc2d3db19513acf331ff08eee47813400ca3d0ed6eaed6a4cd3 WatchSource:0}: Error finding container 14003241d7d50bc2d3db19513acf331ff08eee47813400ca3d0ed6eaed6a4cd3: Status 404 returned error can't find the container with id 14003241d7d50bc2d3db19513acf331ff08eee47813400ca3d0ed6eaed6a4cd3
Sep 18 22:16:49.562125 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.562091    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:16:49.570153 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.570134    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30939455_de05_41c9_8edb_a4b157b28534.slice/crio-a6b17eacc2f8005e6efaeee15970853892250101291c80f4f2edcb18d9a57d84 WatchSource:0}: Error finding container a6b17eacc2f8005e6efaeee15970853892250101291c80f4f2edcb18d9a57d84: Status 404 returned error can't find the container with id a6b17eacc2f8005e6efaeee15970853892250101291c80f4f2edcb18d9a57d84
Sep 18 22:16:49.573679 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.573663    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js"
Sep 18 22:16:49.581492 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.581475    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddb877126_ccbf_4727_a199_5bd4527f08c4.slice/crio-dbce0ce239a7472fb947fbfeb07fc9a7d6e2ba69c93071a05be82b969602368e WatchSource:0}: Error finding container dbce0ce239a7472fb947fbfeb07fc9a7d6e2ba69c93071a05be82b969602368e: Status 404 returned error can't find the container with id dbce0ce239a7472fb947fbfeb07fc9a7d6e2ba69c93071a05be82b969602368e
Sep 18 22:16:49.595531 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.595512    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-psjql"
Sep 18 22:16:49.603903 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.603886    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69d0f323_1928_4b8a_b248_e04808543b76.slice/crio-a14279107aed7701080677ae6b6fea596c084ee3802d6e3c60e5cb76acfe2d9e WatchSource:0}: Error finding container a14279107aed7701080677ae6b6fea596c084ee3802d6e3c60e5cb76acfe2d9e: Status 404 returned error can't find the container with id a14279107aed7701080677ae6b6fea596c084ee3802d6e3c60e5cb76acfe2d9e
Sep 18 22:16:49.612442 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.612425    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-mlkh8"
Sep 18 22:16:49.621252 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:16:49.621232    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1b653c8_c7da_477e_8c16_5885f92b483c.slice/crio-2f35a736d6361f5cf43202a44619d479fbe90bd3e2a214d89df5ce9337bb65e7 WatchSource:0}: Error finding container 2f35a736d6361f5cf43202a44619d479fbe90bd3e2a214d89df5ce9337bb65e7: Status 404 returned error can't find the container with id 2f35a736d6361f5cf43202a44619d479fbe90bd3e2a214d89df5ce9337bb65e7
Sep 18 22:16:49.799984 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.799920    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:49.800109 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.800093    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:49.800170 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.800149    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:16:50.800133743 +0000 UTC m=+2.981718286 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:49.900426 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:49.900331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:49.900583 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.900509    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:49.900583 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:49.900582    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:50.900557776 +0000 UTC m=+3.082142323 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:50.001624 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.001579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:50.001805 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.001736    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:16:50.001805 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.001766    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:16:50.001805 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.001781    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:50.001996 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.001845    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:51.001823856 +0000 UTC m=+3.183408388 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:50.272519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.272432    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 22:11:49 +0000 UTC" deadline="2028-05-15 03:21:25.813968099 +0000 UTC"
Sep 18 22:16:50.272519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.272473    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14501h4m35.541499496s"
Sep 18 22:16:50.297635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.297511    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mlkh8" event={"ID":"a1b653c8-c7da-477e-8c16-5885f92b483c","Type":"ContainerStarted","Data":"2f35a736d6361f5cf43202a44619d479fbe90bd3e2a214d89df5ce9337bb65e7"}
Sep 18 22:16:50.308588 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.308534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjg4b" event={"ID":"30939455-de05-41c9-8edb-a4b157b28534","Type":"ContainerStarted","Data":"a6b17eacc2f8005e6efaeee15970853892250101291c80f4f2edcb18d9a57d84"}
Sep 18 22:16:50.314457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.314421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerStarted","Data":"ed407ed204fd1687d54ff363fb647ee2203512bd0e8a77855cf5e7e030924874"}
Sep 18 22:16:50.317274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.317193    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" event={"ID":"b2d86a8c0ee26cd8a3423d7b77f0c8af","Type":"ContainerStarted","Data":"5f960f0939a57ae02f0dbd8a06093aebb1670ed9f03baa8831d321a7b21ea85b"}
Sep 18 22:16:50.320965 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.320897    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"2d36f920ee1c195e31743d5172b70185","Type":"ContainerStarted","Data":"36859d3767ff86d94b6d32a05cbad0d082680922d29a0ad8ab9b2fec676702b2"}
Sep 18 22:16:50.337104 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.337076    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psjql" event={"ID":"69d0f323-1928-4b8a-b248-e04808543b76","Type":"ContainerStarted","Data":"a14279107aed7701080677ae6b6fea596c084ee3802d6e3c60e5cb76acfe2d9e"}
Sep 18 22:16:50.362435 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.362407    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js" event={"ID":"db877126-ccbf-4727-a199-5bd4527f08c4","Type":"ContainerStarted","Data":"dbce0ce239a7472fb947fbfeb07fc9a7d6e2ba69c93071a05be82b969602368e"}
Sep 18 22:16:50.371749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.371620    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"14003241d7d50bc2d3db19513acf331ff08eee47813400ca3d0ed6eaed6a4cd3"}
Sep 18 22:16:50.385011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.384987    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mbrwj" event={"ID":"c0777467-72d9-4cb1-89de-35f82dc26ec2","Type":"ContainerStarted","Data":"5d55cd36fee548d69b13c81cc8744478f6415c5ee6a6a6256dc227b79f98138f"}
Sep 18 22:16:50.392979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.392925    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jskkx" event={"ID":"c1d01c54-9e81-4b86-b117-fa856a0512e9","Type":"ContainerStarted","Data":"89f79e234b0599d36a0c03261b6ab6876048e1048fb42fbb9e4e30576c60438c"}
Sep 18 22:16:50.418726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.418297    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v7kdp" event={"ID":"ff71cfa3-3462-41c0-93fe-6118322980e0","Type":"ContainerStarted","Data":"82e5488029192e3a7b51497b8170b63cb69e63ad928391335fd3ffcf52dbf54a"}
Sep 18 22:16:50.470971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.470945    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:50.813158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.811483    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:50.813158 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.811765    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:50.813158 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.811840    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:16:52.811817561 +0000 UTC m=+4.993402091 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:50.912920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:50.912879    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:50.913099 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.913044    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:50.913193 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:50.913166    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:52.913142737 +0000 UTC m=+5.094727278 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:51.013881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.013695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:51.014046 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.013925    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:16:51.014046 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.013950    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:16:51.014046 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.013964    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:51.014046 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.014027    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:53.014006493 +0000 UTC m=+5.195591019 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:51.273501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.273408    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-17 22:11:49 +0000 UTC" deadline="2028-03-06 15:13:26.701293992 +0000 UTC"
Sep 18 22:16:51.273501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.273475    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="12832h56m35.427823475s"
Sep 18 22:16:51.285356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.285110    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:51.285356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.285131    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:51.285356 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.285233    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:51.285356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.285133    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:51.285356 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.285317    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:51.285703 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:51.285390    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:16:51.867269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:51.867225    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 18 22:16:52.829159 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:52.829072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:52.829737 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:52.829216    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:52.829737 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:52.829286    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:16:56.829265103 +0000 UTC m=+9.010849644 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:52.930776 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:52.930194    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:52.930776 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:52.930363    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:52.930776 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:52.930428    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:56.930407685 +0000 UTC m=+9.111992213 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:53.031528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:53.030908    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:53.031528 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.031098    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:16:53.031528 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.031118    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:16:53.031528 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.031134    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:53.031528 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.031191    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:16:57.031170115 +0000 UTC m=+9.212754656 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:53.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:53.284859    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:53.285254 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.285010    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:53.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:53.284869    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:53.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:53.284859    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:53.285254 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.285225    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:53.285667 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:53.285333    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:16:55.284842 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:55.284806    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:55.284842 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:55.284830    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:55.285399 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:55.284939    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:55.285399 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:55.285019    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:55.285399 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:55.285094    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:55.285399 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:55.285188    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:16:56.864612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:56.864462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:56.864995 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:56.864613    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:56.864995 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:56.864679    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:17:04.864660207 +0000 UTC m=+17.046244752 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:16:56.964854 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:56.964820    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:56.965018 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:56.964966    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:56.965117 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:56.965025    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:04.965010036 +0000 UTC m=+17.146594560 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:16:57.065769 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:57.065262    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:57.065769 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.065408    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:16:57.065769 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.065428    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:16:57.065769 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.065438    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:57.065769 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.065476    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:05.065464797 +0000 UTC m=+17.247049321 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:16:57.284666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:57.284634    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:57.284849 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.284772    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:57.285237 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:57.285179    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:57.285354 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.285333    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:16:57.285424 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:57.285402    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:57.285498 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:57.285472    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:59.284376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:59.284273    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:16:59.284376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:59.284325    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:16:59.284376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:16:59.284385    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:16:59.284932 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:59.284398    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:16:59.284932 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:59.284490    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:16:59.284932 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:16:59.284605    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:01.284779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:01.284749    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:01.285257 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:01.284833    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:01.285257 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:01.284839    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:01.285257 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:01.284842    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:01.285257 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:01.284911    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:01.285257 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:01.284984    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:03.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:03.285225    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:03.285693 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:03.285225    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:03.285693 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:03.285341    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:03.285693 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:03.285439    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:03.285693 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:03.285228    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:03.285693 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:03.285573    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:04.916307 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:04.916271    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:04.916776 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:04.916428    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:17:04.916776 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:04.916494    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:17:20.916477919 +0000 UTC m=+33.098062448 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:17:05.016922 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:05.016888    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:05.017071 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.017058    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:17:05.017132 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.017121    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:21.017102921 +0000 UTC m=+33.198687447 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:17:05.118182 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:05.118145    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:05.118338 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.118312    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:17:05.118338 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.118330    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:17:05.118338 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.118339    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:17:05.118441 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.118388    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:21.118372401 +0000 UTC m=+33.299956926 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:17:05.284711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:05.284635    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:05.284711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:05.284673    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:05.284878 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.284746    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:05.284878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:05.284800    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:05.284878 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.284863    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:05.284978 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:05.284928    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:07.284744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:07.284709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:07.284744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:07.284728    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:07.285214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:07.284709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:07.285214 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:07.284828    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:07.285214 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:07.284913    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:07.285214 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:07.284985    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.284707    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:09.284949    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.284724    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.284724    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:09.285082    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:09.285227 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:09.285165    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:09.476766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.476735    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" event={"ID":"b2d86a8c0ee26cd8a3423d7b77f0c8af","Type":"ContainerStarted","Data":"b2465263145aaeca098bec2061a8551b43b63b856b7cc0904519d0fdceb93eef"}
Sep 18 22:17:09.482556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.482458    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-psjql" event={"ID":"69d0f323-1928-4b8a-b248-e04808543b76","Type":"ContainerStarted","Data":"5414e8843f354e879007f17612e0c948b06111f4b910cf2d6c8614b52c6710b0"}
Sep 18 22:17:09.491890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.491813    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-129-37.ec2.internal" podStartSLOduration=20.491793389 podStartE2EDuration="20.491793389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:09.490729779 +0000 UTC m=+21.672314326" watchObservedRunningTime="2026-09-18 22:17:09.491793389 +0000 UTC m=+21.673377936"
Sep 18 22:17:09.494926 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.494709    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:17:09.500111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.499686    2551 generic.go:356] "Generic (PLEG): container finished" podID="fcd4e641-bee5-4d87-b8bb-80e070cf7461" containerID="cc80919f1bed0b1d489d58d22b8ba97ba9fcef4633a53202fc8fb56851e221cb" exitCode=1
Sep 18 22:17:09.500111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.499766    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerDied","Data":"cc80919f1bed0b1d489d58d22b8ba97ba9fcef4633a53202fc8fb56851e221cb"}
Sep 18 22:17:09.500111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.499843    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"82173c560f1a273450317e612ef9691369f8552fe7aaf3338ee2b78f4f05f837"}
Sep 18 22:17:09.507406 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.507380    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-jskkx" event={"ID":"c1d01c54-9e81-4b86-b117-fa856a0512e9","Type":"ContainerStarted","Data":"3f6f191f7ccd7fe758f45bfadcab50acdb7c94e41bd0cdeb6cab2a7b44a10314"}
Sep 18 22:17:09.507953 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.507903    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-psjql" podStartSLOduration=1.035199783 podStartE2EDuration="20.507887821s" totalImagesPullingTime="19.472688038s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:09.507136975 +0000 UTC m=+21.688721522" watchObservedRunningTime="2026-09-18 22:17:09.507887821 +0000 UTC m=+21.689472369"
Sep 18 22:17:09.527108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:09.526707    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-jskkx" podStartSLOduration=2.029518091 podStartE2EDuration="21.526672628s" totalImagesPullingTime="19.497154537s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:09.52562948 +0000 UTC m=+21.707214027" watchObservedRunningTime="2026-09-18 22:17:09.526672628 +0000 UTC m=+21.708257175"
Sep 18 22:17:10.512560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.512538    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:17:10.513343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.512997    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"86911b7ef743deb4650752203bfa893ee2de2db871a0c86641eb5ff28120cf75"}
Sep 18 22:17:10.513343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.513107    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"a9c54c992e721ca5e7f5b6b64d16a55c1354d424c8b800eea0b21ac74b69b55a"}
Sep 18 22:17:10.513343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.513162    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"d1d211698d637dcca32fb7815ee22a04e0631ee0affe9a86104eec71a68b488a"}
Sep 18 22:17:10.513343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.513206    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"ceeb78b0ca2fde85bcd3f37c8c1f8bdd64fb42199a885fc6c54583d05bfec80d"}
Sep 18 22:17:10.514943 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.514907    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-mbrwj" event={"ID":"c0777467-72d9-4cb1-89de-35f82dc26ec2","Type":"ContainerStarted","Data":"b78bf548babd49a6658e83e01e29c636452c802b155ceef4381f0328d28bb767"}
Sep 18 22:17:10.516444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.516417    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-v7kdp" event={"ID":"ff71cfa3-3462-41c0-93fe-6118322980e0","Type":"ContainerStarted","Data":"b2961cda269047527ac2194758709f126bf5ca06fbfadda185753b133f7c018b"}
Sep 18 22:17:10.518266 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.518240    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-mlkh8" event={"ID":"a1b653c8-c7da-477e-8c16-5885f92b483c","Type":"ContainerStarted","Data":"0b0974a0f45ee0944f8ba5f93e89d942b32ee14e90b32cece3829e1eb3ffe209"}
Sep 18 22:17:10.520254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.520227    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-pjg4b" event={"ID":"30939455-de05-41c9-8edb-a4b157b28534","Type":"ContainerStarted","Data":"d774ab2aea09d11e889d019a8afa13d588ee8f79815e4c45f15897124093b48b"}
Sep 18 22:17:10.521819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.521793    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="4f65a2e149d9d2fbf67c72d87d69027e0b2a57a9848f9942401625b4962afd2c" exitCode=0
Sep 18 22:17:10.521913 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.521873    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"4f65a2e149d9d2fbf67c72d87d69027e0b2a57a9848f9942401625b4962afd2c"}
Sep 18 22:17:10.523607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.523587    2551 generic.go:356] "Generic (PLEG): container finished" podID="2d36f920ee1c195e31743d5172b70185" containerID="ec204a636403b69274cfda78045fb1c7275224be0c8a8448fae6372d4850a270" exitCode=0
Sep 18 22:17:10.523705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.523652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"2d36f920ee1c195e31743d5172b70185","Type":"ContainerDied","Data":"ec204a636403b69274cfda78045fb1c7275224be0c8a8448fae6372d4850a270"}
Sep 18 22:17:10.525359 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.525295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js" event={"ID":"db877126-ccbf-4727-a199-5bd4527f08c4","Type":"ContainerStarted","Data":"3ddedc7bffb0988d0b70a587a5ed9be08f5130c8267c50596d8535b6ee64d0f6"}
Sep 18 22:17:10.529943 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.529910    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-mbrwj" podStartSLOduration=1.999340031 podStartE2EDuration="21.529896827s" totalImagesPullingTime="19.530556796s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:10.529819654 +0000 UTC m=+22.711404199" watchObservedRunningTime="2026-09-18 22:17:10.529896827 +0000 UTC m=+22.711481376"
Sep 18 22:17:10.563918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.563874    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-pjg4b" podStartSLOduration=2.110629269 podStartE2EDuration="21.563857754s" totalImagesPullingTime="19.453228485s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:10.563614736 +0000 UTC m=+22.745199276" watchObservedRunningTime="2026-09-18 22:17:10.563857754 +0000 UTC m=+22.745442301"
Sep 18 22:17:10.593684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.593644    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-v7kdp" podStartSLOduration=3.026686026 podStartE2EDuration="22.593628885s" totalImagesPullingTime="19.566942859s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:48 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:10.579392637 +0000 UTC m=+22.760977183" watchObservedRunningTime="2026-09-18 22:17:10.593628885 +0000 UTC m=+22.775213431"
Sep 18 22:17:10.594181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.594155    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-mlkh8" podStartSLOduration=2.138925664 podStartE2EDuration="21.594148685s" totalImagesPullingTime="19.455223021s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:10.593620158 +0000 UTC m=+22.775204705" watchObservedRunningTime="2026-09-18 22:17:10.594148685 +0000 UTC m=+22.775733230"
Sep 18 22:17:10.725214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.725183    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:17:10.725750 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:10.725729    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:17:11.188009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.187988    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 18 22:17:11.252093 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.251924    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-18T22:17:11.188005082Z","UUID":"23c646cd-6b65-479e-a8d7-99ddd69c55e0","Handler":null,"Name":"","Endpoint":""}
Sep 18 22:17:11.253275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.253229    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 18 22:17:11.253275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.253263    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 18 22:17:11.284742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.284722    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:11.284828 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.284812    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:11.284915 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:11.284898    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:11.284951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.284816    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:11.284951 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:11.284811    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:11.285016 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:11.284977    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:11.529283 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.529250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js" event={"ID":"db877126-ccbf-4727-a199-5bd4527f08c4","Type":"ContainerStarted","Data":"cacdc9df4eed577145c88962c692c202c58c05dfe293a651f6bd45c32d94f8cb"}
Sep 18 22:17:11.531418 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.531386    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" event={"ID":"2d36f920ee1c195e31743d5172b70185","Type":"ContainerStarted","Data":"902cd1c58cf2995300423217f8ef073bfa57c0fe8691c28851b9f16a2f77a2c0"}
Sep 18 22:17:11.532108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.532066    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:17:11.532377 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.532349    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-pjg4b"
Sep 18 22:17:11.549109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:11.549067    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-129-37.ec2.internal" podStartSLOduration=22.549050474 podStartE2EDuration="22.549050474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:11.547406438 +0000 UTC m=+23.728990983" watchObservedRunningTime="2026-09-18 22:17:11.549050474 +0000 UTC m=+23.730635017"
Sep 18 22:17:12.536012 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:12.535983    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js" event={"ID":"db877126-ccbf-4727-a199-5bd4527f08c4","Type":"ContainerStarted","Data":"d6cab8c9eda10ee073bbc3911dbc02f63b2e77991caca7e4c60a62ebf12716d0"}
Sep 18 22:17:12.539758 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:12.539741    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:17:12.540250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:12.540227    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"2907167b692e71bfa06b11823c27b19df7487dc7fff78088c491055cbc21cd1c"}
Sep 18 22:17:12.554805 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:12.554757    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-sj2js" podStartSLOduration=1.602055201 podStartE2EDuration="23.554738229s" totalImagesPullingTime="21.952683028s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:12.554184396 +0000 UTC m=+24.735768946" watchObservedRunningTime="2026-09-18 22:17:12.554738229 +0000 UTC m=+24.736322777"
Sep 18 22:17:13.285397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:13.285176    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:13.285397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:13.285288    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:13.285397 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:13.285304    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:13.285397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:13.285321    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:13.285809 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:13.285402    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:13.285809 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:13.285472    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:15.284775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:15.284742    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:15.285308 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:15.284744    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:15.285308 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:15.284861    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:15.285308 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:15.284950    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:15.285308 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:15.284745    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:15.285308 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:15.285072    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:16.551777 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.551576    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="b71e0aa158a369419a0d64f0a2a659f35830aa89771288fcd80b6ae7a1b90aef" exitCode=0
Sep 18 22:17:16.552212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.551672    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"b71e0aa158a369419a0d64f0a2a659f35830aa89771288fcd80b6ae7a1b90aef"}
Sep 18 22:17:16.556845 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.556823    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:17:16.557324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.557299    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"6c6e9946cfc2825ec18dc76e8f8c05d89ab048ebadfa1ff23cc315fee7752192"}
Sep 18 22:17:16.557730 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.557703    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:16.557847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.557743    2551 scope.go:117] "RemoveContainer" containerID="cc80919f1bed0b1d489d58d22b8ba97ba9fcef4633a53202fc8fb56851e221cb"
Sep 18 22:17:16.557847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.557785    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:16.575016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:16.574989    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:17.284956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.284858    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:17.285149 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.285002    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:17.285216 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.285195    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:17.285309 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.285285    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:17.285309 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.285300    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:17.285424 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.285404    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:17.542584 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.542517    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w6mzb"]
Sep 18 22:17:17.545314 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.545257    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z8hff"]
Sep 18 22:17:17.557766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.557744    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldvsn"]
Sep 18 22:17:17.565911 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.565895    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:17:17.566320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.566301    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:17.566320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.566303    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" event={"ID":"fcd4e641-bee5-4d87-b8bb-80e070cf7461","Type":"ContainerStarted","Data":"041418962e988c68ccbdbc0bd70cafe3ffcf9f886106367f0d788b7f8545b329"}
Sep 18 22:17:17.566443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.566396    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:17.566443 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.566405    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:17.566513 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.566477    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:17.566513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.566496    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:17.566622 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:17.566602    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:17.566858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.566840    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:17.582433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.582407    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:17.593317 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:17.593278    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q" podStartSLOduration=8.872076481 podStartE2EDuration="28.593262736s" totalImagesPullingTime="19.721186255s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:17.592811705 +0000 UTC m=+29.774396250" watchObservedRunningTime="2026-09-18 22:17:17.593262736 +0000 UTC m=+29.774847278"
Sep 18 22:17:18.570288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:18.570243    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="ad1db97afde2e32da9101392eb6142ca96ed66374dbe7bcf7d0ec373d4d724f4" exitCode=0
Sep 18 22:17:18.570806 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:18.570315    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"ad1db97afde2e32da9101392eb6142ca96ed66374dbe7bcf7d0ec373d4d724f4"}
Sep 18 22:17:19.284922 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:19.284777    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:19.285088 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:19.284825    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:19.285088 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:19.284841    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:19.285088 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:19.284989    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:19.285255 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:19.285107    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:19.285255 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:19.285210    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:20.576536 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:20.576503    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="b46800d1e7d0dd07ce1de7f22a43e9b21cc03099d672cd0bc65b0c2628e3aa0a" exitCode=0
Sep 18 22:17:20.576897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:20.576569    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"b46800d1e7d0dd07ce1de7f22a43e9b21cc03099d672cd0bc65b0c2628e3aa0a"}
Sep 18 22:17:20.933391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:20.933364    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:20.933523 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:20.933468    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 18 22:17:20.933523 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:20.933514    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret podName:881581b3-32aa-456a-8c0f-fa4b9a4f8c6b nodeName:}" failed. No retries permitted until 2026-09-18 22:17:52.933501242 +0000 UTC m=+65.115085772 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret") pod "global-pull-secret-syncer-w6mzb" (UID: "881581b3-32aa-456a-8c0f-fa4b9a4f8c6b") : object "kube-system"/"original-pull-secret" not registered
Sep 18 22:17:21.033637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:21.033609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:21.033753 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.033692    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:17:21.033753 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.033733    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:53.033722298 +0000 UTC m=+65.215306826 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 18 22:17:21.134278 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:21.134253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:21.134373 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.134351    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 18 22:17:21.134373 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.134364    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 18 22:17:21.134373 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.134373    2551 projected.go:196] Error preparing data for projected volume kube-api-access-x5jlq for pod openshift-network-diagnostics/network-check-target-z8hff: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:17:21.134467 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.134415    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq podName:67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9 nodeName:}" failed. No retries permitted until 2026-09-18 22:17:53.134402371 +0000 UTC m=+65.315986898 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-x5jlq" (UniqueName: "kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq") pod "network-check-target-z8hff" (UID: "67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 18 22:17:21.284925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:21.284860    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:21.285062 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:21.284868    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:21.285062 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.284965    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-z8hff" podUID="67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9"
Sep 18 22:17:21.285160 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.285092    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-ldvsn" podUID="e440d077-757a-4d92-a61e-357569199de5"
Sep 18 22:17:21.285160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:21.284869    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:21.285228 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:21.285197    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-w6mzb" podUID="881581b3-32aa-456a-8c0f-fa4b9a4f8c6b"
Sep 18 22:17:22.088752 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.088716    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-129-37.ec2.internal" event="NodeReady"
Sep 18 22:17:22.089147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.088822    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 18 22:17:22.136817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.135154    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:17:22.151303 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.151277    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.154157 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.154101    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x79ck\""
Sep 18 22:17:22.154157 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.154112    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9qkp4"]
Sep 18 22:17:22.154499 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.154477    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 18 22:17:22.154691 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.154675    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 18 22:17:22.159725 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.159706    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 18 22:17:22.164729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.164709    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 18 22:17:22.177239 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.177219    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qkp4"]
Sep 18 22:17:22.177320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.177295    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.181375 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.181356    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 18 22:17:22.181465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.181386    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 18 22:17:22.182374 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.182346    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-lv824\""
Sep 18 22:17:22.187745 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.187726    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 18 22:17:22.192806 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.192788    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:17:22.240865 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.240839    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57160a47-a996-4742-b9e3-7a3d9cd996e9-cert\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.240994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.240888    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.240994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.240911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.240998    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9hnc6\" (UniqueName: \"kubernetes.io/projected/57160a47-a996-4742-b9e3-7a3d9cd996e9-kube-api-access-9hnc6\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.241143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241041    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241124    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241210    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241235    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-njtbt\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.241357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.241342    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.245815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.245791    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-txlsf"]
Sep 18 22:17:22.262088 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.262062    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-txlsf"]
Sep 18 22:17:22.262181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.262169    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.264635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.264616    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 18 22:17:22.264635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.264634    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gtntj\""
Sep 18 22:17:22.264763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.264635    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 18 22:17:22.342625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342569    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-njtbt\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.342625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342598    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.342782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342630    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.342782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342660    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd712ed-b052-43bd-a7b2-691fcb170e5e-tmp-dir\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.342782 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.342733    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:22.342782 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.342750    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:22.342782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342728    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57160a47-a996-4742-b9e3-7a3d9cd996e9-cert\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.343056 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.342891    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:22.842871492 +0000 UTC m=+35.024456041 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:22.343056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342873    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.342982    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9hnc6\" (UniqueName: \"kubernetes.io/projected/57160a47-a996-4742-b9e3-7a3d9cd996e9-kube-api-access-9hnc6\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.343056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343009    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343143    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd712ed-b052-43bd-a7b2-691fcb170e5e-config-volume\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.343299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343169    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8t26f\" (UniqueName: \"kubernetes.io/projected/fcd712ed-b052-43bd-a7b2-691fcb170e5e-kube-api-access-8t26f\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.343299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343199    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343404    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.343773 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.343749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.344685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.344661    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.347393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.347363    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.347491 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.347393    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.347491 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.347415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/57160a47-a996-4742-b9e3-7a3d9cd996e9-cert\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.351481 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.351464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-njtbt\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.352404 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.352382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9hnc6\" (UniqueName: \"kubernetes.io/projected/57160a47-a996-4742-b9e3-7a3d9cd996e9-kube-api-access-9hnc6\") pod \"ingress-canary-9qkp4\" (UID: \"57160a47-a996-4742-b9e3-7a3d9cd996e9\") " pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.352706 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.352682    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.443753 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.443734    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd712ed-b052-43bd-a7b2-691fcb170e5e-config-volume\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.443856 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.443759    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8t26f\" (UniqueName: \"kubernetes.io/projected/fcd712ed-b052-43bd-a7b2-691fcb170e5e-kube-api-access-8t26f\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.443856 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.443780    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.443856 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.443807    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd712ed-b052-43bd-a7b2-691fcb170e5e-tmp-dir\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.443968 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.443898    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:22.443968 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.443956    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:22.943939915 +0000 UTC m=+35.125524442 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:22.444156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.444138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/fcd712ed-b052-43bd-a7b2-691fcb170e5e-tmp-dir\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.444235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.444218    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/fcd712ed-b052-43bd-a7b2-691fcb170e5e-config-volume\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.452949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.452931    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8t26f\" (UniqueName: \"kubernetes.io/projected/fcd712ed-b052-43bd-a7b2-691fcb170e5e-kube-api-access-8t26f\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.486046 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.486011    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9qkp4"
Sep 18 22:17:22.818049 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.817810    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9qkp4"]
Sep 18 22:17:22.822185 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:17:22.822158    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod57160a47_a996_4742_b9e3_7a3d9cd996e9.slice/crio-a323ca2f1b76dcd64c09aef17786eabee8e1613a9cdbe9e73ae31ad32b6a0f17 WatchSource:0}: Error finding container a323ca2f1b76dcd64c09aef17786eabee8e1613a9cdbe9e73ae31ad32b6a0f17: Status 404 returned error can't find the container with id a323ca2f1b76dcd64c09aef17786eabee8e1613a9cdbe9e73ae31ad32b6a0f17
Sep 18 22:17:22.847720 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.847675    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:22.847838 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.847820    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:22.847882 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.847839    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:22.847927 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.847888    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:23.847870944 +0000 UTC m=+36.029455470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:22.948431 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:22.948401    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:22.948598 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.948578    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:22.948663 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:22.948652    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:23.948631932 +0000 UTC m=+36.130216462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:23.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.285227    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:23.285254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.285250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:23.285745 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.285250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:23.290863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.290650    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 22:17:23.290863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.290714    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 22:17:23.291604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.291582    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dq7t2\""
Sep 18 22:17:23.291695 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.291666    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 22:17:23.291882 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.291868    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 22:17:23.291962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.291945    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-95pvs\""
Sep 18 22:17:23.586552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.586478    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qkp4" event={"ID":"57160a47-a996-4742-b9e3-7a3d9cd996e9","Type":"ContainerStarted","Data":"a323ca2f1b76dcd64c09aef17786eabee8e1613a9cdbe9e73ae31ad32b6a0f17"}
Sep 18 22:17:23.855252 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.855171    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:23.855414 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:23.855324    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:23.855414 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:23.855345    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:23.855414 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:23.855409    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:25.85539099 +0000 UTC m=+38.036975514 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:23.956001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:23.955970    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:23.956180 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:23.956150    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:23.956248 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:23.956229    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:25.956206674 +0000 UTC m=+38.137791216 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:25.871384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:25.871337    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:25.871883 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:25.871495    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:25.871883 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:25.871514    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:25.871883 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:25.871581    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:29.871565018 +0000 UTC m=+42.053149548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:25.972041 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:25.971991    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:25.972206 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:25.972172    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:25.972264 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:25.972252    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:29.9722264 +0000 UTC m=+42.153810938 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:26.596784 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:26.596745    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="a810e8116f3554b243f275913cc80bc01b0d09c7dfa499b492a7e5aa079f2873" exitCode=0
Sep 18 22:17:26.596911 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:26.596819    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"a810e8116f3554b243f275913cc80bc01b0d09c7dfa499b492a7e5aa079f2873"}
Sep 18 22:17:26.598422 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:26.598401    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9qkp4" event={"ID":"57160a47-a996-4742-b9e3-7a3d9cd996e9","Type":"ContainerStarted","Data":"b4b7311c66111a61c585dbc918e5ae1a6334970424824cd98383f2ea7c2155ec"}
Sep 18 22:17:27.604198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:27.604169    2551 generic.go:356] "Generic (PLEG): container finished" podID="27cfab9a-6928-47c8-a680-cfbbd86cd311" containerID="5ecd692730fff21203a4826f593a89715d4185a414a6febb302d001ba418889b" exitCode=0
Sep 18 22:17:27.604789 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:27.604253    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerDied","Data":"5ecd692730fff21203a4826f593a89715d4185a414a6febb302d001ba418889b"}
Sep 18 22:17:27.632622 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:27.632489    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9qkp4" podStartSLOduration=2.288787501 podStartE2EDuration="5.632471677s" totalImagesPullingTime="3.343684176s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:26.640359127 +0000 UTC m=+38.821943677" watchObservedRunningTime="2026-09-18 22:17:27.632471677 +0000 UTC m=+39.814056223"
Sep 18 22:17:28.610250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:28.610217    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" event={"ID":"27cfab9a-6928-47c8-a680-cfbbd86cd311","Type":"ContainerStarted","Data":"61ffe748bda10c5d06762ba6a276464ca9e562fb65e8c28ef180833275812fc4"}
Sep 18 22:17:28.633252 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:28.633207    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-sd9wb" podStartSLOduration=6.929499331 podStartE2EDuration="39.63319264s" totalImagesPullingTime="32.703693309s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:28.631702651 +0000 UTC m=+40.813287196" watchObservedRunningTime="2026-09-18 22:17:28.63319264 +0000 UTC m=+40.814777185"
Sep 18 22:17:29.899950 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:29.899915    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:29.900429 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:29.900065    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:29.900429 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:29.900084    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:29.900429 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:29.900158    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:37.900137879 +0000 UTC m=+50.081722430 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:30.001024 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:30.000991    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:30.001176 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:30.001125    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:30.001223 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:30.001192    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:38.001177192 +0000 UTC m=+50.182761715 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:37.953517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:37.953462    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:37.953915 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:37.953640    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:37.953915 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:37.953658    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:37.953915 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:37.953720    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:17:53.95370577 +0000 UTC m=+66.135290298 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:38.054394 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:38.054364    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:38.054498 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:38.054483    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:38.054548 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:38.054535    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:17:54.054522959 +0000 UTC m=+66.236107483 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:49.581574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:49.581548    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-g7g7q"
Sep 18 22:17:52.958194 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:52.958149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:52.961326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:52.961307    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 18 22:17:52.971368 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:52.971343    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/881581b3-32aa-456a-8c0f-fa4b9a4f8c6b-original-pull-secret\") pod \"global-pull-secret-syncer-w6mzb\" (UID: \"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b\") " pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:53.013512 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.013490    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-w6mzb"
Sep 18 22:17:53.059635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.059609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:17:53.062513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.062494    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 18 22:17:53.070802 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:53.070778    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 18 22:17:53.070904 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:53.070860    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs podName:e440d077-757a-4d92-a61e-357569199de5 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:57.070835149 +0000 UTC m=+129.252419695 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs") pod "network-metrics-daemon-ldvsn" (UID: "e440d077-757a-4d92-a61e-357569199de5") : secret "metrics-daemon-secret" not found
Sep 18 22:17:53.160989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.160962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:53.164231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.164215    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 18 22:17:53.174203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.174186    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 18 22:17:53.184419 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.184402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x5jlq\" (UniqueName: \"kubernetes.io/projected/67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9-kube-api-access-x5jlq\") pod \"network-check-target-z8hff\" (UID: \"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9\") " pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:53.281234 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.281157    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-w6mzb"]
Sep 18 22:17:53.284498 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:17:53.284471    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod881581b3_32aa_456a_8c0f_fa4b9a4f8c6b.slice/crio-554c79c19b4b23d22221ffe9fe53582d0688a2816ec7be0f872bfd43808ae8fe WatchSource:0}: Error finding container 554c79c19b4b23d22221ffe9fe53582d0688a2816ec7be0f872bfd43808ae8fe: Status 404 returned error can't find the container with id 554c79c19b4b23d22221ffe9fe53582d0688a2816ec7be0f872bfd43808ae8fe
Sep 18 22:17:53.310214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.310194    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-95pvs\""
Sep 18 22:17:53.318499 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.318481    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:53.586152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.586083    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-z8hff"]
Sep 18 22:17:53.588891 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:17:53.588869    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67bb5038_11f2_46dd_99e6_fc0fe7c8cfb9.slice/crio-404d33d08b4620b762e34eb123765efc035f366a7d89f1a4d9b512be2eda71f0 WatchSource:0}: Error finding container 404d33d08b4620b762e34eb123765efc035f366a7d89f1a4d9b512be2eda71f0: Status 404 returned error can't find the container with id 404d33d08b4620b762e34eb123765efc035f366a7d89f1a4d9b512be2eda71f0
Sep 18 22:17:53.669215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.669186    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z8hff" event={"ID":"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9","Type":"ContainerStarted","Data":"404d33d08b4620b762e34eb123765efc035f366a7d89f1a4d9b512be2eda71f0"}
Sep 18 22:17:53.670334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.670311    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w6mzb" event={"ID":"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b","Type":"ContainerStarted","Data":"554c79c19b4b23d22221ffe9fe53582d0688a2816ec7be0f872bfd43808ae8fe"}
Sep 18 22:17:53.965122 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:53.965086    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:17:53.965562 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:53.965275    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 18 22:17:53.965562 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:53.965296    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-7db79cc748-mmxw5: secret "image-registry-tls" not found
Sep 18 22:17:53.965562 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:53.965371    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls podName:f853784f-d81b-486a-b65c-4e715b7b615f nodeName:}" failed. No retries permitted until 2026-09-18 22:18:25.965347869 +0000 UTC m=+98.146932409 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls") pod "image-registry-7db79cc748-mmxw5" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f") : secret "image-registry-tls" not found
Sep 18 22:17:54.066158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:54.066123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:17:54.066312 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:54.066293    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 18 22:17:54.066381 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:17:54.066369    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls podName:fcd712ed-b052-43bd-a7b2-691fcb170e5e nodeName:}" failed. No retries permitted until 2026-09-18 22:18:26.066348132 +0000 UTC m=+98.247932669 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls") pod "dns-default-txlsf" (UID: "fcd712ed-b052-43bd-a7b2-691fcb170e5e") : secret "dns-default-metrics-tls" not found
Sep 18 22:17:54.920820 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:54.920785    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"]
Sep 18 22:17:54.923073 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:54.923055    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"
Sep 18 22:17:54.925682 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:54.925661    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-89wd6\""
Sep 18 22:17:54.937513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:54.937490    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"]
Sep 18 22:17:55.074672 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.074636    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j9nfw\" (UniqueName: \"kubernetes.io/projected/da8220e7-39dc-42e4-880b-8a17277551cc-kube-api-access-j9nfw\") pod \"network-check-source-7cbc45fddc-jx88m\" (UID: \"da8220e7-39dc-42e4-880b-8a17277551cc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"
Sep 18 22:17:55.175518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.175433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j9nfw\" (UniqueName: \"kubernetes.io/projected/da8220e7-39dc-42e4-880b-8a17277551cc-kube-api-access-j9nfw\") pod \"network-check-source-7cbc45fddc-jx88m\" (UID: \"da8220e7-39dc-42e4-880b-8a17277551cc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"
Sep 18 22:17:55.185546 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.185495    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j9nfw\" (UniqueName: \"kubernetes.io/projected/da8220e7-39dc-42e4-880b-8a17277551cc-kube-api-access-j9nfw\") pod \"network-check-source-7cbc45fddc-jx88m\" (UID: \"da8220e7-39dc-42e4-880b-8a17277551cc\") " pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"
Sep 18 22:17:55.232112 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.232086    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"
Sep 18 22:17:55.509839 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.509737    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m"]
Sep 18 22:17:55.514191 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:17:55.514153    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda8220e7_39dc_42e4_880b_8a17277551cc.slice/crio-af08427618eb7eb14c4d741245b2695f1255f4a3f7b2da68bff615f6429996ab WatchSource:0}: Error finding container af08427618eb7eb14c4d741245b2695f1255f4a3f7b2da68bff615f6429996ab: Status 404 returned error can't find the container with id af08427618eb7eb14c4d741245b2695f1255f4a3f7b2da68bff615f6429996ab
Sep 18 22:17:55.677714 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:55.677676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m" event={"ID":"da8220e7-39dc-42e4-880b-8a17277551cc","Type":"ContainerStarted","Data":"af08427618eb7eb14c4d741245b2695f1255f4a3f7b2da68bff615f6429996ab"}
Sep 18 22:17:58.687521 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.687484    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m" event={"ID":"da8220e7-39dc-42e4-880b-8a17277551cc","Type":"ContainerStarted","Data":"fbf8d67e2759bbdcfd273f1be537d0c7b206e315a402cb1bd072c409d3d806a4"}
Sep 18 22:17:58.689154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.689131    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-z8hff" event={"ID":"67bb5038-11f2-46dd-99e6-fc0fe7c8cfb9","Type":"ContainerStarted","Data":"d1a535ba6471b46f5dabd090ad99af0def663ae7a128b3485a0af88b7ee75662"}
Sep 18 22:17:58.689301 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.689180    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:17:58.690734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.690717    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-w6mzb" event={"ID":"881581b3-32aa-456a-8c0f-fa4b9a4f8c6b","Type":"ContainerStarted","Data":"53f0658f05fe9233c4d17875909d0041a88ba1acffa8959d02c70622930b6a5b"}
Sep 18 22:17:58.704896 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.704837    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-7cbc45fddc-jx88m" podStartSLOduration=1.679254687 podStartE2EDuration="4.704818609s" totalImagesPullingTime="3.025563922s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:17:54 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:58.703671566 +0000 UTC m=+70.885256111" watchObservedRunningTime="2026-09-18 22:17:58.704818609 +0000 UTC m=+70.886403159"
Sep 18 22:17:58.727347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.727306    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-z8hff" podStartSLOduration=64.769138029 podStartE2EDuration="1m9.727290878s" totalImagesPullingTime="4.958152849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:58.726673412 +0000 UTC m=+70.908257958" watchObservedRunningTime="2026-09-18 22:17:58.727290878 +0000 UTC m=+70.908875424"
Sep 18 22:17:58.744914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:17:58.744869    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-w6mzb" podStartSLOduration=64.490361149 podStartE2EDuration="1m9.744852018s" totalImagesPullingTime="5.254490869s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:17:58.74379641 +0000 UTC m=+70.925380955" watchObservedRunningTime="2026-09-18 22:17:58.744852018 +0000 UTC m=+70.926436563"
Sep 18 22:18:00.166985 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.166947    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-4tsqh"]
Sep 18 22:18:00.184396 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.184363    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.187986 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.187959    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 18 22:18:00.188483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.188465    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-7xc5k\""
Sep 18 22:18:00.188559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.188506    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 18 22:18:00.188618 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.188551    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 18 22:18:00.193615 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.193591    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 18 22:18:00.195816 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.195793    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tsqh"]
Sep 18 22:18:00.205023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.205000    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"]
Sep 18 22:18:00.212467 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.212449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43098664-d625-4793-8ac6-dfadfead8f26-crio-socket\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.212531 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.212482    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.212531 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.212503    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43098664-d625-4793-8ac6-dfadfead8f26-data-volume\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.212531 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.212518    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qhzcj\" (UniqueName: \"kubernetes.io/projected/43098664-d625-4793-8ac6-dfadfead8f26-kube-api-access-qhzcj\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.212641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.212591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.225606 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.225588    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"]
Sep 18 22:18:00.225687 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.225680    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"
Sep 18 22:18:00.228435 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.228411    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-8cf8j\""
Sep 18 22:18:00.228681 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.228665    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Sep 18 22:18:00.228926 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.228905    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Sep 18 22:18:00.312861 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312831    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7h64j\" (UniqueName: \"kubernetes.io/projected/92223b3f-6257-4aea-aad6-283fc83005bf-kube-api-access-7h64j\") pod \"migrator-76794cbd6c-52kqb\" (UID: \"92223b3f-6257-4aea-aad6-283fc83005bf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312895    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43098664-d625-4793-8ac6-dfadfead8f26-crio-socket\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43098664-d625-4793-8ac6-dfadfead8f26-data-volume\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312967    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/43098664-d625-4793-8ac6-dfadfead8f26-crio-socket\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.312972    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qhzcj\" (UniqueName: \"kubernetes.io/projected/43098664-d625-4793-8ac6-dfadfead8f26-kube-api-access-qhzcj\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313061 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.313051    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 22:18:00.313363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.313099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.313363 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.313116    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:00.813096706 +0000 UTC m=+72.994681248 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : configmap "kube-rbac-proxy" not found
Sep 18 22:18:00.313363 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.313174    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:00.313363 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.313230    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:00.813214656 +0000 UTC m=+72.994799184 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:00.313363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.313351    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/43098664-d625-4793-8ac6-dfadfead8f26-data-volume\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.335369 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.335346    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qhzcj\" (UniqueName: \"kubernetes.io/projected/43098664-d625-4793-8ac6-dfadfead8f26-kube-api-access-qhzcj\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.413721 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.413679    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7h64j\" (UniqueName: \"kubernetes.io/projected/92223b3f-6257-4aea-aad6-283fc83005bf-kube-api-access-7h64j\") pod \"migrator-76794cbd6c-52kqb\" (UID: \"92223b3f-6257-4aea-aad6-283fc83005bf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"
Sep 18 22:18:00.423149 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.423089    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7h64j\" (UniqueName: \"kubernetes.io/projected/92223b3f-6257-4aea-aad6-283fc83005bf-kube-api-access-7h64j\") pod \"migrator-76794cbd6c-52kqb\" (UID: \"92223b3f-6257-4aea-aad6-283fc83005bf\") " pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"
Sep 18 22:18:00.533535 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.533500    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"
Sep 18 22:18:00.817248 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.817169    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.817248 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.817230    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:00.817408 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.817301    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 22:18:00.817408 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.817334    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:00.817408 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.817371    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:01.817354223 +0000 UTC m=+73.998938761 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : configmap "kube-rbac-proxy" not found
Sep 18 22:18:00.817408 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:00.817386    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:01.817379947 +0000 UTC m=+73.998964470 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:00.821877 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:00.821850    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb"]
Sep 18 22:18:00.824893 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:00.824870    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92223b3f_6257_4aea_aad6_283fc83005bf.slice/crio-cefb6a0af2c99dc7d1dfe5c1b8c12c681b94df66b059674f80311a70a926e7d0 WatchSource:0}: Error finding container cefb6a0af2c99dc7d1dfe5c1b8c12c681b94df66b059674f80311a70a926e7d0: Status 404 returned error can't find the container with id cefb6a0af2c99dc7d1dfe5c1b8c12c681b94df66b059674f80311a70a926e7d0
Sep 18 22:18:01.700220 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:01.700184    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb" event={"ID":"92223b3f-6257-4aea-aad6-283fc83005bf","Type":"ContainerStarted","Data":"cefb6a0af2c99dc7d1dfe5c1b8c12c681b94df66b059674f80311a70a926e7d0"}
Sep 18 22:18:01.826069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:01.826006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:01.826233 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:01.826095    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:01.826233 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:01.826141    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 22:18:01.826233 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:01.826226    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:03.826207596 +0000 UTC m=+76.007792123 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : configmap "kube-rbac-proxy" not found
Sep 18 22:18:01.826233 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:01.826225    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:01.826368 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:01.826256    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:03.826249214 +0000 UTC m=+76.007833738 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:02.311569 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:02.311542    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v7kdp_ff71cfa3-3462-41c0-93fe-6118322980e0/dns-node-resolver/0.log"
Sep 18 22:18:02.706359 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:02.706328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb" event={"ID":"92223b3f-6257-4aea-aad6-283fc83005bf","Type":"ContainerStarted","Data":"42ff31ea5a4a41913ddc8de5b694d6b4df986b0aa05b0af9802746b2cafd572c"}
Sep 18 22:18:02.706719 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:02.706378    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb" event={"ID":"92223b3f-6257-4aea-aad6-283fc83005bf","Type":"ContainerStarted","Data":"0bc1ee5fc10d8e07476d3a3ef208059484297f29b70c08e945d6156968cb14e0"}
Sep 18 22:18:02.724759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:02.724713    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-76794cbd6c-52kqb" podStartSLOduration=1.38534275 podStartE2EDuration="2.724697342s" totalImagesPullingTime="1.339354592s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:02.72402246 +0000 UTC m=+74.905607007" watchObservedRunningTime="2026-09-18 22:18:02.724697342 +0000 UTC m=+74.906281888"
Sep 18 22:18:03.312523 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:03.312497    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mbrwj_c0777467-72d9-4cb1-89de-35f82dc26ec2/node-ca/0.log"
Sep 18 22:18:03.839690 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:03.839638    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:03.840100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:03.839712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:03.840100 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:03.839766    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 22:18:03.840100 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:03.839817    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:03.840100 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:03.839849    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:07.839832282 +0000 UTC m=+80.021416806 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : configmap "kube-rbac-proxy" not found
Sep 18 22:18:03.840100 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:03.839868    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:07.839855719 +0000 UTC m=+80.021440271 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:04.113241 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:04.113138    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qkp4_57160a47-a996-4742-b9e3-7a3d9cd996e9/serve-healthcheck-canary/0.log"
Sep 18 22:18:07.872197 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:07.872145    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:07.872657 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:07.872229    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:07.872657 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:07.872266    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 18 22:18:07.872657 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:07.872338    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:07.872657 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:07.872344    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:15.872328396 +0000 UTC m=+88.053912919 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : configmap "kube-rbac-proxy" not found
Sep 18 22:18:07.872657 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:07.872379    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls podName:43098664-d625-4793-8ac6-dfadfead8f26 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:15.872366293 +0000 UTC m=+88.053950817 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls") pod "insights-runtime-extractor-4tsqh" (UID: "43098664-d625-4793-8ac6-dfadfead8f26") : secret "insights-runtime-extractor-tls" not found
Sep 18 22:18:15.925958 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:15.925801    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:15.926436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:15.925974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:15.926503 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:15.926476    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/43098664-d625-4793-8ac6-dfadfead8f26-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:15.928179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:15.928159    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/43098664-d625-4793-8ac6-dfadfead8f26-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-4tsqh\" (UID: \"43098664-d625-4793-8ac6-dfadfead8f26\") " pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:16.094814 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:16.094781    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-4tsqh"
Sep 18 22:18:16.369147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:16.369075    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-4tsqh"]
Sep 18 22:18:16.372796 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:16.372771    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43098664_d625_4793_8ac6_dfadfead8f26.slice/crio-0f2eda765999f1815f3526284fff7dc791b4500da43d8b47d453517dd5d681ee WatchSource:0}: Error finding container 0f2eda765999f1815f3526284fff7dc791b4500da43d8b47d453517dd5d681ee: Status 404 returned error can't find the container with id 0f2eda765999f1815f3526284fff7dc791b4500da43d8b47d453517dd5d681ee
Sep 18 22:18:16.752011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:16.751977    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tsqh" event={"ID":"43098664-d625-4793-8ac6-dfadfead8f26","Type":"ContainerStarted","Data":"dc260eb0e5e044ad534007ddd6959fce2205605486202584531d9131db7ef317"}
Sep 18 22:18:16.752157 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:16.752054    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tsqh" event={"ID":"43098664-d625-4793-8ac6-dfadfead8f26","Type":"ContainerStarted","Data":"0f2eda765999f1815f3526284fff7dc791b4500da43d8b47d453517dd5d681ee"}
Sep 18 22:18:17.757676 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:17.757636    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tsqh" event={"ID":"43098664-d625-4793-8ac6-dfadfead8f26","Type":"ContainerStarted","Data":"af76526a845b0bd4aa0e88fc216a2c1a1c9240ed595d0693818a5c788c1a003b"}
Sep 18 22:18:18.763236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:18.763159    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-4tsqh" event={"ID":"43098664-d625-4793-8ac6-dfadfead8f26","Type":"ContainerStarted","Data":"f5e28b6438384ec74cf51c8bd2577b1eccd8bed360adbe231123ae9e82d6bafe"}
Sep 18 22:18:18.781621 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:18.781554    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-4tsqh" podStartSLOduration=16.76038315 podStartE2EDuration="18.781539563s" totalImagesPullingTime="2.021156413s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:18:00 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:18.78119969 +0000 UTC m=+90.962784235" watchObservedRunningTime="2026-09-18 22:18:18.781539563 +0000 UTC m=+90.963124108"
Sep 18 22:18:23.044638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.044597    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"]
Sep 18 22:18:23.047793 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.047769    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.051830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.051807    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 18 22:18:23.052792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.052774    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 18 22:18:23.052919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.052807    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 18 22:18:23.053095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.053077    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 18 22:18:23.055651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.055624    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"]
Sep 18 22:18:23.058668 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.058649    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.063582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.063553    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-console\"/\"networking-console-plugin\""
Sep 18 22:18:23.063791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.063772    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"]
Sep 18 22:18:23.066911 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.066896    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"networking-console-plugin-cert\""
Sep 18 22:18:23.067481 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.067466    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-console\"/\"default-dockercfg-hnf6x\""
Sep 18 22:18:23.080545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.080523    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sz9jv\" (UniqueName: \"kubernetes.io/projected/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-kube-api-access-sz9jv\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.080729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.080596    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.080729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.080678    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.080729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.080704    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-tmp\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.080729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.080720    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.086532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.086511    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"]
Sep 18 22:18:23.154373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.154353    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6594f4c75-v9jlt"]
Sep 18 22:18:23.157191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.157176    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:23.160169 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.160152    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Sep 18 22:18:23.160555 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.160539    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Sep 18 22:18:23.160981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.160965    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"default-dockercfg-x8q5s\""
Sep 18 22:18:23.177641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.177621    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-v9jlt"]
Sep 18 22:18:23.181909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.181890    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.182001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.181921    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psksx\" (UniqueName: \"kubernetes.io/projected/1435f7fe-3c57-4700-ac27-f9fac9f5d622-kube-api-access-psksx\") pod \"downloads-6594f4c75-v9jlt\" (UID: \"1435f7fe-3c57-4700-ac27-f9fac9f5d622\") " pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:23.182001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.181962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.182001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.181982    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-tmp\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.182177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.182089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.182364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.182341    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sz9jv\" (UniqueName: \"kubernetes.io/projected/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-kube-api-access-sz9jv\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.182473 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.182370    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-tmp\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.182645 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.182624    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-nginx-conf\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.184509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.184480    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"networking-console-plugin-cert\" (UniqueName: \"kubernetes.io/secret/5be494a2-c1ec-497f-b8ab-42a9d0b9529b-networking-console-plugin-cert\") pod \"networking-console-plugin-6446c8d75d-jmzd9\" (UID: \"5be494a2-c1ec-497f-b8ab-42a9d0b9529b\") " pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.184509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.184498    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-klusterlet-config\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.197273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.197250    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sz9jv\" (UniqueName: \"kubernetes.io/projected/efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7-kube-api-access-sz9jv\") pod \"klusterlet-addon-workmgr-7b5dc56d76-945d7\" (UID: \"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.282956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.282923    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psksx\" (UniqueName: \"kubernetes.io/projected/1435f7fe-3c57-4700-ac27-f9fac9f5d622-kube-api-access-psksx\") pod \"downloads-6594f4c75-v9jlt\" (UID: \"1435f7fe-3c57-4700-ac27-f9fac9f5d622\") " pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:23.298400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.298353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-psksx\" (UniqueName: \"kubernetes.io/projected/1435f7fe-3c57-4700-ac27-f9fac9f5d622-kube-api-access-psksx\") pod \"downloads-6594f4c75-v9jlt\" (UID: \"1435f7fe-3c57-4700-ac27-f9fac9f5d622\") " pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:23.356345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.356316    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:23.367317 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.367293    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"
Sep 18 22:18:23.465956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.465924    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:23.626540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.626465    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"]
Sep 18 22:18:23.629534 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:23.629505    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefce5a33_abb0_4ab8_8ca6_7fa3ff24e0c7.slice/crio-d4cc467fb61daf710f61ad9af6a710bca6c2aea89917a3414f34b7f4cfaeb1cb WatchSource:0}: Error finding container d4cc467fb61daf710f61ad9af6a710bca6c2aea89917a3414f34b7f4cfaeb1cb: Status 404 returned error can't find the container with id d4cc467fb61daf710f61ad9af6a710bca6c2aea89917a3414f34b7f4cfaeb1cb
Sep 18 22:18:23.649603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.649580    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9"]
Sep 18 22:18:23.651613 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:23.651592    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5be494a2_c1ec_497f_b8ab_42a9d0b9529b.slice/crio-3d09e96e97431b3a07c5db47fd7595bf4dd835a6e71bb2d4a293c6a90c99a361 WatchSource:0}: Error finding container 3d09e96e97431b3a07c5db47fd7595bf4dd835a6e71bb2d4a293c6a90c99a361: Status 404 returned error can't find the container with id 3d09e96e97431b3a07c5db47fd7595bf4dd835a6e71bb2d4a293c6a90c99a361
Sep 18 22:18:23.733517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.733491    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6594f4c75-v9jlt"]
Sep 18 22:18:23.736473 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:23.736448    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1435f7fe_3c57_4700_ac27_f9fac9f5d622.slice/crio-17ec2570a76197ce5a6ddeeb4bb0f2bae5cdadcb7d3110d442d67ec5c28d344e WatchSource:0}: Error finding container 17ec2570a76197ce5a6ddeeb4bb0f2bae5cdadcb7d3110d442d67ec5c28d344e: Status 404 returned error can't find the container with id 17ec2570a76197ce5a6ddeeb4bb0f2bae5cdadcb7d3110d442d67ec5c28d344e
Sep 18 22:18:23.781124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.781085    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9" event={"ID":"5be494a2-c1ec-497f-b8ab-42a9d0b9529b","Type":"ContainerStarted","Data":"3d09e96e97431b3a07c5db47fd7595bf4dd835a6e71bb2d4a293c6a90c99a361"}
Sep 18 22:18:23.782253 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.782223    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7" event={"ID":"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7","Type":"ContainerStarted","Data":"d4cc467fb61daf710f61ad9af6a710bca6c2aea89917a3414f34b7f4cfaeb1cb"}
Sep 18 22:18:23.783267 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:23.783246    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-v9jlt" event={"ID":"1435f7fe-3c57-4700-ac27-f9fac9f5d622","Type":"ContainerStarted","Data":"17ec2570a76197ce5a6ddeeb4bb0f2bae5cdadcb7d3110d442d67ec5c28d344e"}
Sep 18 22:18:25.794847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:25.794516    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9" event={"ID":"5be494a2-c1ec-497f-b8ab-42a9d0b9529b","Type":"ContainerStarted","Data":"4a80b2891fba4ea01b505b4b2aefa981909807f60c7ec4455d1db98095da6a67"}
Sep 18 22:18:26.003396 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.003319    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:18:26.011290 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.010750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"image-registry-7db79cc748-mmxw5\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") " pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:18:26.066241 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.065955    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-x79ck\""
Sep 18 22:18:26.072433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.072401    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:18:26.104745 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.104287    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:18:26.107743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.107644    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/fcd712ed-b052-43bd-a7b2-691fcb170e5e-metrics-tls\") pod \"dns-default-txlsf\" (UID: \"fcd712ed-b052-43bd-a7b2-691fcb170e5e\") " pod="openshift-dns/dns-default-txlsf"
Sep 18 22:18:26.174084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.174060    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-gtntj\""
Sep 18 22:18:26.180603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.180577    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-txlsf"
Sep 18 22:18:26.473638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.473572    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-console/networking-console-plugin-6446c8d75d-jmzd9" podStartSLOduration=2.080852936 podStartE2EDuration="3.473545698s" totalImagesPullingTime="1.392692762s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:25.811469615 +0000 UTC m=+97.993054161" watchObservedRunningTime="2026-09-18 22:18:26.473545698 +0000 UTC m=+98.655130244"
Sep 18 22:18:26.474051 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.474015    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-txlsf"]
Sep 18 22:18:26.477655 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:26.477613    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfcd712ed_b052_43bd_a7b2_691fcb170e5e.slice/crio-1533ded605d3a1b251e8a8c4c1a29979fda4bb416d439ca4b4f21599f6aadab8 WatchSource:0}: Error finding container 1533ded605d3a1b251e8a8c4c1a29979fda4bb416d439ca4b4f21599f6aadab8: Status 404 returned error can't find the container with id 1533ded605d3a1b251e8a8c4c1a29979fda4bb416d439ca4b4f21599f6aadab8
Sep 18 22:18:26.481444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.481382    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:18:26.484981 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:26.484941    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf853784f_d81b_486a_b65c_4e715b7b615f.slice/crio-b2cc742a8e47e24766eb029ed736cda48876e09d5fe1b634a051a12a55dfdc27 WatchSource:0}: Error finding container b2cc742a8e47e24766eb029ed736cda48876e09d5fe1b634a051a12a55dfdc27: Status 404 returned error can't find the container with id b2cc742a8e47e24766eb029ed736cda48876e09d5fe1b634a051a12a55dfdc27
Sep 18 22:18:26.801637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.801522    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txlsf" event={"ID":"fcd712ed-b052-43bd-a7b2-691fcb170e5e","Type":"ContainerStarted","Data":"1533ded605d3a1b251e8a8c4c1a29979fda4bb416d439ca4b4f21599f6aadab8"}
Sep 18 22:18:26.803363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.803337    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" event={"ID":"f853784f-d81b-486a-b65c-4e715b7b615f","Type":"ContainerStarted","Data":"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"}
Sep 18 22:18:26.803474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.803382    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" event={"ID":"f853784f-d81b-486a-b65c-4e715b7b615f","Type":"ContainerStarted","Data":"b2cc742a8e47e24766eb029ed736cda48876e09d5fe1b634a051a12a55dfdc27"}
Sep 18 22:18:26.827270 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:26.827131    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" podStartSLOduration=97.827114922 podStartE2EDuration="1m37.827114922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:26.825543633 +0000 UTC m=+99.007128179" watchObservedRunningTime="2026-09-18 22:18:26.827114922 +0000 UTC m=+99.008699467"
Sep 18 22:18:27.278973 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.278912    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:18:27.286547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.286523    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.289232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.289203    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-oauth-config\""
Sep 18 22:18:27.291331 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.290981    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"service-ca\""
Sep 18 22:18:27.291331 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.291073    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"oauth-serving-cert\""
Sep 18 22:18:27.291331 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.290986    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"console-config\""
Sep 18 22:18:27.291547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.291445    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-serving-cert\""
Sep 18 22:18:27.291655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.291634    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"console-dockercfg-tz6wt\""
Sep 18 22:18:27.292005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.291951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:18:27.316821 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.316785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.316978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.316831    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.316978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.316862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6lfhq\" (UniqueName: \"kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.316978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.316924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.317164 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.316977    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.317164 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.317005    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418060    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6lfhq\" (UniqueName: \"kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418113    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418316 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418287    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418379    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.418488 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.418417    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.419226 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.419179    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.419332 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.419255    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.419404 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.419381    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.421271 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.421248    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.421372 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.421276    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.427342 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.427299    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6lfhq\" (UniqueName: \"kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq\") pod \"console-66b4567478-s5ftl\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") " pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.600965 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.600932    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:27.807679 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:27.807606    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:18:28.498099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.497998    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:18:28.525503 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:28.525468    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1337c01a_47b9_4514_8937_b60a822593c4.slice/crio-eb2380dfab299c7019109209df570366c6e58007f01e58306ebebc29b6c69d12 WatchSource:0}: Error finding container eb2380dfab299c7019109209df570366c6e58007f01e58306ebebc29b6c69d12: Status 404 returned error can't find the container with id eb2380dfab299c7019109209df570366c6e58007f01e58306ebebc29b6c69d12
Sep 18 22:18:28.813162 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.813071    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b4567478-s5ftl" event={"ID":"1337c01a-47b9-4514-8937-b60a822593c4","Type":"ContainerStarted","Data":"eb2380dfab299c7019109209df570366c6e58007f01e58306ebebc29b6c69d12"}
Sep 18 22:18:28.815288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.815258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7" event={"ID":"efce5a33-abb0-4ab8-8ca6-7fa3ff24e0c7","Type":"ContainerStarted","Data":"d8cd6b4e56030069f28a430ba2834863016b31d454b8d661ce46b6d19095209a"}
Sep 18 22:18:28.815592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.815563    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:28.817239 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.817190    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7"
Sep 18 22:18:28.835846 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:28.835796    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7b5dc56d76-945d7" podStartSLOduration=0.904246894 podStartE2EDuration="5.835779744s" totalImagesPullingTime="4.93153285s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:28.834346786 +0000 UTC m=+101.015931333" watchObservedRunningTime="2026-09-18 22:18:28.835779744 +0000 UTC m=+101.017364290"
Sep 18 22:18:29.696519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:29.696485    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-z8hff"
Sep 18 22:18:32.467744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.467696    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"]
Sep 18 22:18:32.470322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.470299    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.473067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.472854    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-dockercfg-2x87p\""
Sep 18 22:18:32.473067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.472938    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 18 22:18:32.474072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.474025    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 18 22:18:32.474563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.474269    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-kube-rbac-proxy-config\""
Sep 18 22:18:32.474563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.474458    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 18 22:18:32.474563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.474486    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-tls\""
Sep 18 22:18:32.484140 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.484116    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"]
Sep 18 22:18:32.563909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.563875    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a39227a-ff22-4ef1-b877-2f2fa394474b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.564085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.563996    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.564085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.564060    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.564204 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.564092    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6cft2\" (UniqueName: \"kubernetes.io/projected/3a39227a-ff22-4ef1-b877-2f2fa394474b-kube-api-access-6cft2\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.665372 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.665331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a39227a-ff22-4ef1-b877-2f2fa394474b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.665559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.665413    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.665559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.665440    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.665559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.665471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6cft2\" (UniqueName: \"kubernetes.io/projected/3a39227a-ff22-4ef1-b877-2f2fa394474b-kube-api-access-6cft2\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.666177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.666152    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/3a39227a-ff22-4ef1-b877-2f2fa394474b-metrics-client-ca\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.668475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.668432    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-kube-rbac-proxy-config\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.669480 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.669456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-operator-tls\" (UniqueName: \"kubernetes.io/secret/3a39227a-ff22-4ef1-b877-2f2fa394474b-prometheus-operator-tls\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.673911 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.673887    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6cft2\" (UniqueName: \"kubernetes.io/projected/3a39227a-ff22-4ef1-b877-2f2fa394474b-kube-api-access-6cft2\") pod \"prometheus-operator-7bd5d56d6-4qn5j\" (UID: \"3a39227a-ff22-4ef1-b877-2f2fa394474b\") " pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:32.782636 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:32.782546    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"
Sep 18 22:18:33.839543 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:33.839503    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b4567478-s5ftl" event={"ID":"1337c01a-47b9-4514-8937-b60a822593c4","Type":"ContainerStarted","Data":"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"}
Sep 18 22:18:33.841215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:33.841188    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txlsf" event={"ID":"fcd712ed-b052-43bd-a7b2-691fcb170e5e","Type":"ContainerStarted","Data":"9530808de55cf36ec4e02fa1b24de989d5ce725ebc9a2ffc3d4f9f080ff0b79c"}
Sep 18 22:18:33.859239 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:33.859185    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/console-66b4567478-s5ftl" podStartSLOduration=1.737412653 podStartE2EDuration="6.859166243s" totalImagesPullingTime="5.12175359s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:33.858618993 +0000 UTC m=+106.040203569" watchObservedRunningTime="2026-09-18 22:18:33.859166243 +0000 UTC m=+106.040750788"
Sep 18 22:18:33.892140 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:33.892054    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j"]
Sep 18 22:18:33.895276 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:33.895241    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3a39227a_ff22_4ef1_b877_2f2fa394474b.slice/crio-e2c01e153ba23c2e3d9e872c280f50dd610bcea8d1863ea093b196f598dcca00 WatchSource:0}: Error finding container e2c01e153ba23c2e3d9e872c280f50dd610bcea8d1863ea093b196f598dcca00: Status 404 returned error can't find the container with id e2c01e153ba23c2e3d9e872c280f50dd610bcea8d1863ea093b196f598dcca00
Sep 18 22:18:34.847915 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:34.847877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-txlsf" event={"ID":"fcd712ed-b052-43bd-a7b2-691fcb170e5e","Type":"ContainerStarted","Data":"f0c77a7ac08cec1fb789da3466e03d7409a87c6ebe1ab2d00e4d08b368202f5e"}
Sep 18 22:18:34.848381 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:34.847987    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-txlsf"
Sep 18 22:18:34.850226 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:34.850190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j" event={"ID":"3a39227a-ff22-4ef1-b877-2f2fa394474b","Type":"ContainerStarted","Data":"e2c01e153ba23c2e3d9e872c280f50dd610bcea8d1863ea093b196f598dcca00"}
Sep 18 22:18:37.601695 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:37.601662    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:37.602163 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:37.601768    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:18:37.603414 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:37.603375    2551 patch_prober.go:28] interesting pod/console-66b4567478-s5ftl container/console namespace/openshift-console: Startup probe status=failure output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused" start-of-body=
Sep 18 22:18:37.603532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:37.603438    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-console/console-66b4567478-s5ftl" podUID="1337c01a-47b9-4514-8937-b60a822593c4" containerName="console" probeResult="failure" output="Get \"https://10.132.0.15:8443/health\": dial tcp 10.132.0.15:8443: connect: connection refused"
Sep 18 22:18:42.882749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.882710    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6594f4c75-v9jlt" event={"ID":"1435f7fe-3c57-4700-ac27-f9fac9f5d622","Type":"ContainerStarted","Data":"95184514e7ef3f57a49ca87ac03ac289267b716e314fe36e733886b33f3e5bb4"}
Sep 18 22:18:42.883245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.882922    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:42.885705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.885675    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j" event={"ID":"3a39227a-ff22-4ef1-b877-2f2fa394474b","Type":"ContainerStarted","Data":"c2ad9952f4d38013b85090895a4c89250642fbe88ebc93dd1d8cc7982f7c3aca"}
Sep 18 22:18:42.885831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.885730    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j" event={"ID":"3a39227a-ff22-4ef1-b877-2f2fa394474b","Type":"ContainerStarted","Data":"c6f5579e8decc28759fdb0f14bfa62ff6ffc77d043ce163b364159203c5f38ec"}
Sep 18 22:18:42.892711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.892686    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6594f4c75-v9jlt"
Sep 18 22:18:42.899708 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.899659    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-txlsf" podStartSLOduration=73.770510565 podStartE2EDuration="1m20.899639081s" totalImagesPullingTime="7.129128516s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:17:22 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:34.865642202 +0000 UTC m=+107.047226750" watchObservedRunningTime="2026-09-18 22:18:42.899639081 +0000 UTC m=+115.081223629"
Sep 18 22:18:42.900270 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.900230    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-console/downloads-6594f4c75-v9jlt" podStartSLOduration=1.479422938 podStartE2EDuration="19.900219335s" totalImagesPullingTime="18.420796397s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:23 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:42.899920094 +0000 UTC m=+115.081504639" watchObservedRunningTime="2026-09-18 22:18:42.900219335 +0000 UTC m=+115.081803881"
Sep 18 22:18:42.919685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:42.919632    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-7bd5d56d6-4qn5j" podStartSLOduration=2.684800811 podStartE2EDuration="10.919616335s" totalImagesPullingTime="8.234815524s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:42.9186099 +0000 UTC m=+115.100194446" watchObservedRunningTime="2026-09-18 22:18:42.919616335 +0000 UTC m=+115.101200880"
Sep 18 22:18:44.808461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.808423    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"]
Sep 18 22:18:44.840989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.840961    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"]
Sep 18 22:18:44.841186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.841009    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-d9tq6"]
Sep 18 22:18:44.841186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.841131    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:44.843507 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.843478    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Sep 18 22:18:44.843649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.843575    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-bnphc\""
Sep 18 22:18:44.843649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.843598    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Sep 18 22:18:44.854951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.854927    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.856412 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.856391    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-txlsf"
Sep 18 22:18:44.857228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.857209    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 18 22:18:44.857334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.857302    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 18 22:18:44.857555 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.857532    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-825pr\""
Sep 18 22:18:44.857655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.857532    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 18 22:18:44.976656 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.976615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jtkhp\" (UniqueName: \"kubernetes.io/projected/dfb09f48-c756-4259-86ff-2395ef6b9c06-kube-api-access-jtkhp\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:44.976826 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.976677    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb09f48-c756-4259-86ff-2395ef6b9c06-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:44.976892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.976870    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-textfile\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.976953 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.976935    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:44.977005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.976979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977093 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977024    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-root\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977159    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977206    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-sys\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977243    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:44.977320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977266    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-wtmp\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-metrics-client-ca\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977377    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.977559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.977478    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5h2q\" (UniqueName: \"kubernetes.io/projected/602ce902-1acc-4181-8da1-93fff9d8c6e5-kube-api-access-d5h2q\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:44.982727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.982705    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:18:44.987752 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.987725    2551 patch_prober.go:28] interesting pod/image-registry-7db79cc748-mmxw5 container/registry namespace/openshift-image-registry: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body={"errors":[{"code":"UNAVAILABLE","message":"service unavailable","detail":"health check failed: please see /debug/health"}]}
Sep 18 22:18:44.987843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:44.987786    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" containerName="registry" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:18:45.078855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078766    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.078855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-root\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.078855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078848    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078871    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-sys\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078920    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-wtmp\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.078980    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-sys\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-root\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079054    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-metrics-client-ca\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079117    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5h2q\" (UniqueName: \"kubernetes.io/projected/602ce902-1acc-4181-8da1-93fff9d8c6e5-kube-api-access-d5h2q\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079117    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-wtmp\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jtkhp\" (UniqueName: \"kubernetes.io/projected/dfb09f48-c756-4259-86ff-2395ef6b9c06-kube-api-access-jtkhp\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb09f48-c756-4259-86ff-2395ef6b9c06-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:45.079206    2551 secret.go:189] Couldn't get secret openshift-monitoring/node-exporter-tls: secret "node-exporter-tls" not found
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079223    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-textfile\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:45.079274    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls podName:602ce902-1acc-4181-8da1-93fff9d8c6e5 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:45.579249223 +0000 UTC m=+117.760833753 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "node-exporter-tls" (UniqueName: "kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls") pod "node-exporter-d9tq6" (UID: "602ce902-1acc-4181-8da1-93fff9d8c6e5") : secret "node-exporter-tls" not found
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:45.079423    2551 secret.go:189] Couldn't get secret openshift-monitoring/openshift-state-metrics-tls: secret "openshift-state-metrics-tls" not found
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:45.079462    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls podName:dfb09f48-c756-4259-86ff-2395ef6b9c06 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:45.579449237 +0000 UTC m=+117.761033764 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "openshift-state-metrics-tls" (UniqueName: "kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls") pod "openshift-state-metrics-86c5bc79d6-qx8x7" (UID: "dfb09f48-c756-4259-86ff-2395ef6b9c06") : secret "openshift-state-metrics-tls" not found
Sep 18 22:18:45.079637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079555    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-accelerators-collector-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.080062 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.079858    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-textfile\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.080062 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.080003    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/602ce902-1acc-4181-8da1-93fff9d8c6e5-metrics-client-ca\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.080170 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.080154    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/dfb09f48-c756-4259-86ff-2395ef6b9c06-metrics-client-ca\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.081977 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.081952    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.082109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.081958    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.091234 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.091210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5h2q\" (UniqueName: \"kubernetes.io/projected/602ce902-1acc-4181-8da1-93fff9d8c6e5-kube-api-access-d5h2q\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.092059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.092006    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jtkhp\" (UniqueName: \"kubernetes.io/projected/dfb09f48-c756-4259-86ff-2395ef6b9c06-kube-api-access-jtkhp\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.582444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.582406    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.582625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.582489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.585167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.585141    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/602ce902-1acc-4181-8da1-93fff9d8c6e5-node-exporter-tls\") pod \"node-exporter-d9tq6\" (UID: \"602ce902-1acc-4181-8da1-93fff9d8c6e5\") " pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.585279 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.585206    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/dfb09f48-c756-4259-86ff-2395ef6b9c06-openshift-state-metrics-tls\") pod \"openshift-state-metrics-86c5bc79d6-qx8x7\" (UID: \"dfb09f48-c756-4259-86ff-2395ef6b9c06\") " pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.752615 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.752582    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"
Sep 18 22:18:45.764784 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.764752    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-d9tq6"
Sep 18 22:18:45.874408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.874173    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 22:18:45.906323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.906191    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9tq6" event={"ID":"602ce902-1acc-4181-8da1-93fff9d8c6e5","Type":"ContainerStarted","Data":"d5441b468b4982725f648a323f92565584c85057017dafebf1cba27b96a47cd2"}
Sep 18 22:18:45.906323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.906250    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 22:18:45.906555 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.906439    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.909333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.909310    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-cluster-tls-config\""
Sep 18 22:18:45.909888 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.909867    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy\""
Sep 18 22:18:45.910144 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.909914    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-web\""
Sep 18 22:18:45.910144 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.910065    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-generated\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.910326    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-web-config\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.910419    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-dockercfg-jhvcv\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.910829    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls-assets-0\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.911307    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-kube-rbac-proxy-metric\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.911503    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"alertmanager-trusted-ca-bundle\""
Sep 18 22:18:45.912398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.910379    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"alertmanager-main-tls\""
Sep 18 22:18:45.986454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986417    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986456    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986489    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986600    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-77rnp\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-kube-api-access-77rnp\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986632    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986677    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-out\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986830    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986844    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-web-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:45.986881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:45.986863    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.074211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.074152    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7"]
Sep 18 22:18:46.077125 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:46.077091    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddfb09f48_c756_4259_86ff_2395ef6b9c06.slice/crio-01116c29148f20e8b2b5019efaf77d33f1cb078b0a987bdc3718d0f71962c7ed WatchSource:0}: Error finding container 01116c29148f20e8b2b5019efaf77d33f1cb078b0a987bdc3718d0f71962c7ed: Status 404 returned error can't find the container with id 01116c29148f20e8b2b5019efaf77d33f1cb078b0a987bdc3718d0f71962c7ed
Sep 18 22:18:46.081779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.081755    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:18:46.087851 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.087828    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.087960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.087858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.087960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.087884    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.087960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.087912    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.087960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.087942    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-77rnp\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-kube-api-access-77rnp\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088181 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:46.088021    2551 secret.go:189] Couldn't get secret openshift-monitoring/alertmanager-main-tls: secret "alertmanager-main-tls" not found
Sep 18 22:18:46.088181 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:46.088100    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls podName:983b7fc0-60e4-4ca0-8980-4526551a0cc0 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:46.588080071 +0000 UTC m=+118.769664609 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "secret-alertmanager-main-tls" (UniqueName: "kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls") pod "alertmanager-main-0" (UID: "983b7fc0-60e4-4ca0-8980-4526551a0cc0") : secret "alertmanager-main-tls" not found
Sep 18 22:18:46.088181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088130    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088170    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-out\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088265    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088300    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088324    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-web-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088379    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.088775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.088749    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-main-db\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-main-db\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.089129 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.089108    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-metrics-client-ca\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.091654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.091277    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-tls-assets\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.091654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.091362    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-out\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.092090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.091838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.092090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.091902    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-metric\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-metric\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.092090 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:18:46.091942    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle podName:983b7fc0-60e4-4ca0-8980-4526551a0cc0 nodeName:}" failed. No retries permitted until 2026-09-18 22:18:46.591925926 +0000 UTC m=+118.773510451 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "alertmanager-trusted-ca-bundle" (UniqueName: "kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle") pod "alertmanager-main-0" (UID: "983b7fc0-60e4-4ca0-8980-4526551a0cc0") : configmap references non-existent config key: ca-bundle.crt
Sep 18 22:18:46.093018 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.092967    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-config-volume\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.093130 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.093090    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-kube-rbac-proxy-web\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.094602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.094571    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cluster-tls-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-cluster-tls-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.095725 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.095684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-web-config\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.106595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.106569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-77rnp\" (UniqueName: \"kubernetes.io/projected/983b7fc0-60e4-4ca0-8980-4526551a0cc0-kube-api-access-77rnp\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.593084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.593049    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.593279 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.593140    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.593994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.593964    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"alertmanager-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/983b7fc0-60e4-4ca0-8980-4526551a0cc0-alertmanager-trusted-ca-bundle\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.595668 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.595646    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-alertmanager-main-tls\" (UniqueName: \"kubernetes.io/secret/983b7fc0-60e4-4ca0-8980-4526551a0cc0-secret-alertmanager-main-tls\") pod \"alertmanager-main-0\" (UID: \"983b7fc0-60e4-4ca0-8980-4526551a0cc0\") " pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.819726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.819689    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/alertmanager-main-0"
Sep 18 22:18:46.905238 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.905199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7" event={"ID":"dfb09f48-c756-4259-86ff-2395ef6b9c06","Type":"ContainerStarted","Data":"7bba252a2a13fede149dd9a5b828da1f23bd5fc41297403ce81552caa092723b"}
Sep 18 22:18:46.905627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.905264    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7" event={"ID":"dfb09f48-c756-4259-86ff-2395ef6b9c06","Type":"ContainerStarted","Data":"207b7bb658a424b496ae5c642becf4766644394d0c34a4e9e70ff5ce78ed7fd0"}
Sep 18 22:18:46.905627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:46.905285    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7" event={"ID":"dfb09f48-c756-4259-86ff-2395ef6b9c06","Type":"ContainerStarted","Data":"01116c29148f20e8b2b5019efaf77d33f1cb078b0a987bdc3718d0f71962c7ed"}
Sep 18 22:18:47.249115 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:47.249088    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/alertmanager-main-0"]
Sep 18 22:18:47.304307 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:47.304267    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod983b7fc0_60e4_4ca0_8980_4526551a0cc0.slice/crio-789b909531b287e6c325c8eefb03f9e797bb3f797db6a0e22f0ba96f4405f6e6 WatchSource:0}: Error finding container 789b909531b287e6c325c8eefb03f9e797bb3f797db6a0e22f0ba96f4405f6e6: Status 404 returned error can't find the container with id 789b909531b287e6c325c8eefb03f9e797bb3f797db6a0e22f0ba96f4405f6e6
Sep 18 22:18:47.912093 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:47.911850    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"789b909531b287e6c325c8eefb03f9e797bb3f797db6a0e22f0ba96f4405f6e6"}
Sep 18 22:18:47.914237 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:47.914206    2551 generic.go:356] "Generic (PLEG): container finished" podID="602ce902-1acc-4181-8da1-93fff9d8c6e5" containerID="20b413a7fd4bda724fa6514aa4f350ea036e4945f26e6681abf04cc902bbf85a" exitCode=0
Sep 18 22:18:47.914363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:47.914274    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9tq6" event={"ID":"602ce902-1acc-4181-8da1-93fff9d8c6e5","Type":"ContainerDied","Data":"20b413a7fd4bda724fa6514aa4f350ea036e4945f26e6681abf04cc902bbf85a"}
Sep 18 22:18:48.921974 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:48.921928    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7" event={"ID":"dfb09f48-c756-4259-86ff-2395ef6b9c06","Type":"ContainerStarted","Data":"f85cd808bfa90e1d4a66cf0fa41765b85a0bdc8bde9764ca97c3a13193cab3c6"}
Sep 18 22:18:48.925273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:48.925244    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9tq6" event={"ID":"602ce902-1acc-4181-8da1-93fff9d8c6e5","Type":"ContainerStarted","Data":"12b5ac0cbe9b77d9c12be70958d01badbc2d9b9e1ac9f8380c7d6640c6a7b2d1"}
Sep 18 22:18:48.925273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:48.925300    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-d9tq6" event={"ID":"602ce902-1acc-4181-8da1-93fff9d8c6e5","Type":"ContainerStarted","Data":"0a71835f258bcd153ab356e4f21d12cffdd01111bb7350d83d5a531c3f7bfd79"}
Sep 18 22:18:48.945278 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:48.945222    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-86c5bc79d6-qx8x7" podStartSLOduration=2.886338446 podStartE2EDuration="4.945205915s" totalImagesPullingTime="2.058867469s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:48.943206674 +0000 UTC m=+121.124791222" watchObservedRunningTime="2026-09-18 22:18:48.945205915 +0000 UTC m=+121.126790462"
Sep 18 22:18:48.972149 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:48.969674    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-d9tq6" podStartSLOduration=3.371123489 podStartE2EDuration="4.969655114s" totalImagesPullingTime="1.501684293s" totalInitContainerRuntime="96.847332ms" isStatefulPod=true podCreationTimestamp="2026-09-18 22:18:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:48.963007206 +0000 UTC m=+121.144591752" watchObservedRunningTime="2026-09-18 22:18:48.969655114 +0000 UTC m=+121.151239661"
Sep 18 22:18:49.931530 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:49.931465    2551 generic.go:356] "Generic (PLEG): container finished" podID="983b7fc0-60e4-4ca0-8980-4526551a0cc0" containerID="e7a20772d5b796bf9cb487cb854148f859bba6cd71f1beee927fc93a19b1a66d" exitCode=0
Sep 18 22:18:49.931890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:49.931547    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerDied","Data":"e7a20772d5b796bf9cb487cb854148f859bba6cd71f1beee927fc93a19b1a66d"}
Sep 18 22:18:51.223259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.221904    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 22:18:51.250482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.250268    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 22:18:51.250482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.250441    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.254382    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.254670    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-63g2efg06sviq\""
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.254992    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.255348    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-p99t2\""
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.255627    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Sep 18 22:18:51.256158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.255811    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Sep 18 22:18:51.256649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.256632    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Sep 18 22:18:51.256714 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.256652    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Sep 18 22:18:51.260068 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.259872    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Sep 18 22:18:51.261021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.260495    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Sep 18 22:18:51.261021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.260748    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Sep 18 22:18:51.261021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.260942    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Sep 18 22:18:51.261391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.261160    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Sep 18 22:18:51.261391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.261347    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Sep 18 22:18:51.261699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.261526    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Sep 18 22:18:51.268726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.267999    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Sep 18 22:18:51.270513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.270495    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Sep 18 22:18:51.338197 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvmts\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-kube-api-access-hvmts\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338444    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338491    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338517    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338567    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338747 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338836    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338855    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.338892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338877    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.338980    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339049    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339094    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339160    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339354 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339216    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339354 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.339354 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.339295    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440468 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440468 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440470    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440653    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvmts\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-kube-api-access-hvmts\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440729    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440860 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440771    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440860 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440802    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.440860 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440834    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440873    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440880    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440940    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.440974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441005    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441051    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441076    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441102    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441150    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441189    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441216    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441240    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441271    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441714 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441296    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.441714 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.441425    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.445667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.444046    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.450904 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.450463    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.451638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.451612    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.452075 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.452051    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.452448 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.452424    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.455042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.454997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.455330 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.455309    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.456000 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.455853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463066 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.461955    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463066 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.462415    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463066 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.462979    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463359 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.463286    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-config-out\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463617 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.463594    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-web-config\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.463920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.463898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.464130 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.464109    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.464515 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.464451    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.464800 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.464779    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvmts\" (UniqueName: \"kubernetes.io/projected/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-kube-api-access-hvmts\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.466376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.466320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6\") " pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.572841 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.572714    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:18:51.946644 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:51.946615    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Sep 18 22:18:51.974725 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:51.974700    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda06ca16d_bb05_43dd_ae67_c7f6e40f1cf6.slice/crio-48fe91a9cda18f147ad878e2591eec1ca0ec09f7aebb1281d1b168ba7081b318 WatchSource:0}: Error finding container 48fe91a9cda18f147ad878e2591eec1ca0ec09f7aebb1281d1b168ba7081b318: Status 404 returned error can't find the container with id 48fe91a9cda18f147ad878e2591eec1ca0ec09f7aebb1281d1b168ba7081b318
Sep 18 22:18:52.950972 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.950940    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"aba57db5295297d181d775b250625dec50ed79bfe94caf3d7cc83ffa983d7bed"}
Sep 18 22:18:52.951432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.951009    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"3ec61138088fdf2a5780efaa701191ed58b69dd67eed553d3a3496a7573a9d7a"}
Sep 18 22:18:52.951432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.951055    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"66d6c322fd4bb98b34b75efff3cada062b7359a60c5ec033b64fd3309e3d0619"}
Sep 18 22:18:52.951432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.951098    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"d8d35dd5e0934b2cae608109a28f7d8fe3d3075d37cad293fa09532b38064bb1"}
Sep 18 22:18:52.951432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.951128    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"d30f82d9cc181a4bc6f2e0e78b0cb2e914cc7e5abd406d30f43c382ef6355d12"}
Sep 18 22:18:52.952849 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.952818    2551 generic.go:356] "Generic (PLEG): container finished" podID="a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6" containerID="ca7991d6af9228050aadaf0eeeb1d66d5d873fe07156f418369abdbaff44b554" exitCode=0
Sep 18 22:18:52.952970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.952924    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerDied","Data":"ca7991d6af9228050aadaf0eeeb1d66d5d873fe07156f418369abdbaff44b554"}
Sep 18 22:18:52.953052 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:52.952994    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"48fe91a9cda18f147ad878e2591eec1ca0ec09f7aebb1281d1b168ba7081b318"}
Sep 18 22:18:53.961704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:53.961618    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/alertmanager-main-0" event={"ID":"983b7fc0-60e4-4ca0-8980-4526551a0cc0","Type":"ContainerStarted","Data":"95959a7e17d3a1f94d4deea2b07a811fba0915cb7792fbb79eaf001d41c8e313"}
Sep 18 22:18:53.994063 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:53.994004    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/alertmanager-main-0" podStartSLOduration=3.395082398 podStartE2EDuration="8.993983353s" totalImagesPullingTime="5.576203199s" totalInitContainerRuntime="22.697756ms" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:45 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:18:53.993287069 +0000 UTC m=+126.174871631" watchObservedRunningTime="2026-09-18 22:18:53.993983353 +0000 UTC m=+126.175567902"
Sep 18 22:18:54.989723 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:54.989438    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:18:56.977641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:56.977604    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"17bba2fb4ea9130d29e01f21db346ba4e6ce37fd97fa23f20e19d7dcc67f198e"}
Sep 18 22:18:57.092336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.092302    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:18:57.095281 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.095249    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/e440d077-757a-4d92-a61e-357569199de5-metrics-certs\") pod \"network-metrics-daemon-ldvsn\" (UID: \"e440d077-757a-4d92-a61e-357569199de5\") " pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:18:57.202173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.202147    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-dq7t2\""
Sep 18 22:18:57.210568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.210543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-ldvsn"
Sep 18 22:18:57.492485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.492459    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-ldvsn"]
Sep 18 22:18:57.495009 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:18:57.494976    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode440d077_757a_4d92_a61e_357569199de5.slice/crio-023e34354ea1de98463a1659089aacf6ee2d96e6ba10ff95001d8be9697e7ae8 WatchSource:0}: Error finding container 023e34354ea1de98463a1659089aacf6ee2d96e6ba10ff95001d8be9697e7ae8: Status 404 returned error can't find the container with id 023e34354ea1de98463a1659089aacf6ee2d96e6ba10ff95001d8be9697e7ae8
Sep 18 22:18:57.985777 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.985740    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"9184a0b6c96f2ded9cd883d551a542fc905c7741dd3e51e2409f5ca21125f8f1"}
Sep 18 22:18:57.987163 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:18:57.987133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldvsn" event={"ID":"e440d077-757a-4d92-a61e-357569199de5","Type":"ContainerStarted","Data":"023e34354ea1de98463a1659089aacf6ee2d96e6ba10ff95001d8be9697e7ae8"}
Sep 18 22:19:00.001103 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:00.001066    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldvsn" event={"ID":"e440d077-757a-4d92-a61e-357569199de5","Type":"ContainerStarted","Data":"878667b0251825ddd88102aefd3943178f1d373d1ff690e1b1488f8798cb751a"}
Sep 18 22:19:00.004187 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:00.004129    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"7bb64f7e74ea6d41a2b079188c26a407a3a02292ee4329bb1035942be8e5b239"}
Sep 18 22:19:00.004304 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:00.004211    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"64280fbef618b4b7cdf83fd495bd8ea472a5258ec69e1e4249016ffc8a538dc8"}
Sep 18 22:19:01.012689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.012643    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"94d5d5becf58dab1dd2758030fc0fa8ba40abda5e3e459a3e1e3f2136d5d9bb0"}
Sep 18 22:19:01.013165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.012740    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6","Type":"ContainerStarted","Data":"a1da9daa1fc8a6522b431755bdcab66f6f2afa582726e8d1249c83e561b1f50f"}
Sep 18 22:19:01.014454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.014427    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-ldvsn" event={"ID":"e440d077-757a-4d92-a61e-357569199de5","Type":"ContainerStarted","Data":"9562447ecc8da53d3d939e47b383493ba9345db07e9b8d82ef5634c49d89ec7a"}
Sep 18 22:19:01.044278 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.044222    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.553832017 podStartE2EDuration="10.044203457s" totalImagesPullingTime="6.470727301s" totalInitContainerRuntime="19.644139ms" isStatefulPod=false podCreationTimestamp="2026-09-18 22:18:51 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:19:01.04240168 +0000 UTC m=+133.223986225" watchObservedRunningTime="2026-09-18 22:19:01.044203457 +0000 UTC m=+133.225788002"
Sep 18 22:19:01.061974 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.061927    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-ldvsn" podStartSLOduration=129.860920677 podStartE2EDuration="2m12.061913292s" totalImagesPullingTime="2.200992615s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:16:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:19:01.060542489 +0000 UTC m=+133.242127035" watchObservedRunningTime="2026-09-18 22:19:01.061913292 +0000 UTC m=+133.243497841"
Sep 18 22:19:01.573762 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:01.573719    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:19:10.004655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.004585    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" containerName="registry" containerID="cri-o://e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b" gracePeriod=30
Sep 18 22:19:10.276721 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.276642    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:19:10.305545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305420    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305483    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305509    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-njtbt\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305537    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305664    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305743    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305873    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.305949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305939    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration\") pod \"f853784f-d81b-486a-b65c-4e715b7b615f\" (UID: \"f853784f-d81b-486a-b65c-4e715b7b615f\") "
Sep 18 22:19:10.306178 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.305992    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:19:10.306844 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.306334    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-trusted-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.306844 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.306404    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:19:10.308849 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.308806    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls" (OuterVolumeSpecName: "registry-tls") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "registry-tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:19:10.308849 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.308825    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:19:10.309048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.309006    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:19:10.309171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.309063    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt" (OuterVolumeSpecName: "kube-api-access-njtbt") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "kube-api-access-njtbt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:19:10.316325 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.316297    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:19:10.317228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.317202    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f853784f-d81b-486a-b65c-4e715b7b615f" (UID: "f853784f-d81b-486a-b65c-4e715b7b615f"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:19:10.407554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407510    2551 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-bound-sa-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407545    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-njtbt\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-kube-api-access-njtbt\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407555    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f853784f-d81b-486a-b65c-4e715b7b615f-registry-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407563    2551 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f853784f-d81b-486a-b65c-4e715b7b615f-ca-trust-extracted\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407571    2551 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f853784f-d81b-486a-b65c-4e715b7b615f-registry-certificates\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407581    2551 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-image-registry-private-configuration\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:10.407799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:10.407589    2551 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f853784f-d81b-486a-b65c-4e715b7b615f-installation-pull-secrets\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.054470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.054432    2551 generic.go:356] "Generic (PLEG): container finished" podID="f853784f-d81b-486a-b65c-4e715b7b615f" containerID="e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b" exitCode=0
Sep 18 22:19:11.054843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.054480    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5"
Sep 18 22:19:11.054843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.054521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" event={"ID":"f853784f-d81b-486a-b65c-4e715b7b615f","Type":"ContainerDied","Data":"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"}
Sep 18 22:19:11.054843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.054577    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-7db79cc748-mmxw5" event={"ID":"f853784f-d81b-486a-b65c-4e715b7b615f","Type":"ContainerDied","Data":"b2cc742a8e47e24766eb029ed736cda48876e09d5fe1b634a051a12a55dfdc27"}
Sep 18 22:19:11.054843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.054620    2551 scope.go:117] "RemoveContainer" containerID="e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"
Sep 18 22:19:11.073886 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.073866    2551 scope.go:117] "RemoveContainer" containerID="e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"
Sep 18 22:19:11.074221 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:19:11.074201    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b\": container with ID starting with e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b not found: ID does not exist" containerID="e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"
Sep 18 22:19:11.074285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.074229    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b"} err="failed to get container status \"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b\": rpc error: code = NotFound desc = could not find container \"e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b\": container with ID starting with e7fcf3a71735d9e26a52fd7b2fd51ade0c73193007d125054ff3cc55524c1f7b not found: ID does not exist"
Sep 18 22:19:11.086442 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.086414    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:19:11.091136 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.091115    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-7db79cc748-mmxw5"]
Sep 18 22:19:11.137652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.137603    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-console/console-66b4567478-s5ftl" podUID="1337c01a-47b9-4514-8937-b60a822593c4" containerName="console" containerID="cri-o://d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb" gracePeriod=15
Sep 18 22:19:11.404181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.404152    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:19:11.518329 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518298    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6lfhq\" (UniqueName: \"kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518329 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518343    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518363    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518387    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518561    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518629    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config\") pod \"1337c01a-47b9-4514-8937-b60a822593c4\" (UID: \"1337c01a-47b9-4514-8937-b60a822593c4\") "
Sep 18 22:19:11.518867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518802    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca" (OuterVolumeSpecName: "service-ca") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:19:11.518867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518812    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert" (OuterVolumeSpecName: "oauth-serving-cert") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "oauth-serving-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:19:11.519074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518986    2551 reconciler_common.go:299] "Volume detached for volume \"oauth-serving-cert\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-oauth-serving-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.519074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.519005    2551 reconciler_common.go:299] "Volume detached for volume \"service-ca\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.519074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.518981    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config" (OuterVolumeSpecName: "console-config") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "console-config". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:19:11.520631 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.520608    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config" (OuterVolumeSpecName: "console-oauth-config") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "console-oauth-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:19:11.520966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.520943    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert" (OuterVolumeSpecName: "console-serving-cert") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "console-serving-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:19:11.521048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.520965    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq" (OuterVolumeSpecName: "kube-api-access-6lfhq") pod "1337c01a-47b9-4514-8937-b60a822593c4" (UID: "1337c01a-47b9-4514-8937-b60a822593c4"). InnerVolumeSpecName "kube-api-access-6lfhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:19:11.619470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.619375    2551 reconciler_common.go:299] "Volume detached for volume \"console-serving-cert\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-serving-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.619470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.619408    2551 reconciler_common.go:299] "Volume detached for volume \"console-config\" (UniqueName: \"kubernetes.io/configmap/1337c01a-47b9-4514-8937-b60a822593c4-console-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.619470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.619422    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6lfhq\" (UniqueName: \"kubernetes.io/projected/1337c01a-47b9-4514-8937-b60a822593c4-kube-api-access-6lfhq\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:11.619470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:11.619434    2551 reconciler_common.go:299] "Volume detached for volume \"console-oauth-config\" (UniqueName: \"kubernetes.io/secret/1337c01a-47b9-4514-8937-b60a822593c4-console-oauth-config\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:19:12.059417 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.059378    2551 generic.go:356] "Generic (PLEG): container finished" podID="1337c01a-47b9-4514-8937-b60a822593c4" containerID="d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb" exitCode=0
Sep 18 22:19:12.059853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.059428    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/console-66b4567478-s5ftl"
Sep 18 22:19:12.059853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.059468    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b4567478-s5ftl" event={"ID":"1337c01a-47b9-4514-8937-b60a822593c4","Type":"ContainerDied","Data":"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"}
Sep 18 22:19:12.059853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.059558    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-console/console-66b4567478-s5ftl" event={"ID":"1337c01a-47b9-4514-8937-b60a822593c4","Type":"ContainerDied","Data":"eb2380dfab299c7019109209df570366c6e58007f01e58306ebebc29b6c69d12"}
Sep 18 22:19:12.059853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.059587    2551 scope.go:117] "RemoveContainer" containerID="d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"
Sep 18 22:19:12.068436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.068286    2551 scope.go:117] "RemoveContainer" containerID="d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"
Sep 18 22:19:12.068637 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:19:12.068616    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb\": container with ID starting with d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb not found: ID does not exist" containerID="d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"
Sep 18 22:19:12.068709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.068650    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb"} err="failed to get container status \"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb\": rpc error: code = NotFound desc = could not find container \"d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb\": container with ID starting with d18280d2de048350d3188dc85632e6dc4d272a6be86866cf66f4247c85cb54fb not found: ID does not exist"
Sep 18 22:19:12.081325 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.081293    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:19:12.084226 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.084199    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-console/console-66b4567478-s5ftl"]
Sep 18 22:19:12.290400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.290370    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1337c01a-47b9-4514-8937-b60a822593c4" path="/var/lib/kubelet/pods/1337c01a-47b9-4514-8937-b60a822593c4/volumes"
Sep 18 22:19:12.290781 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:12.290768    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" path="/var/lib/kubelet/pods/f853784f-d81b-486a-b65c-4e715b7b615f/volumes"
Sep 18 22:19:51.573547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:51.573503    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:19:51.594880 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:51.594848    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:19:52.233771 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:19:52.233736    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Sep 18 22:21:48.202383 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:48.202354    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:21:48.205386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:48.205359    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:21:48.208823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:48.208803    2551 kubelet.go:1648] "Image garbage collection succeeded"
Sep 18 22:21:56.783083 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783042    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"]
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783370    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" containerName="registry"
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783385    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" containerName="registry"
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783409    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1337c01a-47b9-4514-8937-b60a822593c4" containerName="console"
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783415    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1337c01a-47b9-4514-8937-b60a822593c4" containerName="console"
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783480    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1337c01a-47b9-4514-8937-b60a822593c4" containerName="console"
Sep 18 22:21:56.784496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.783490    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f853784f-d81b-486a-b65c-4e715b7b615f" containerName="registry"
Sep 18 22:21:56.785323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.785306    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:56.787903 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.787880    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"openshift-service-ca.crt\""
Sep 18 22:21:56.788824 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.788805    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"openshift-lws-operator-dockercfg-4mwbz\""
Sep 18 22:21:56.788922 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.788855    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"kube-root-ca.crt\""
Sep 18 22:21:56.795098 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.795073    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"]
Sep 18 22:21:56.907204 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.907172    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88e16948-5195-4cab-88ae-6a129e4a3214-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:56.907402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:56.907216    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5qxdq\" (UniqueName: \"kubernetes.io/projected/88e16948-5195-4cab-88ae-6a129e4a3214-kube-api-access-5qxdq\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.008632 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.008600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5qxdq\" (UniqueName: \"kubernetes.io/projected/88e16948-5195-4cab-88ae-6a129e4a3214-kube-api-access-5qxdq\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.008779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.008700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88e16948-5195-4cab-88ae-6a129e4a3214-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.009143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.009125    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/88e16948-5195-4cab-88ae-6a129e4a3214-tmp\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.019461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.019428    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5qxdq\" (UniqueName: \"kubernetes.io/projected/88e16948-5195-4cab-88ae-6a129e4a3214-kube-api-access-5qxdq\") pod \"openshift-lws-operator-fd8ccff4c-7n77f\" (UID: \"88e16948-5195-4cab-88ae-6a129e4a3214\") " pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.103343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.103241    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"
Sep 18 22:21:57.374322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.374242    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f"]
Sep 18 22:21:57.377782 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:21:57.377758    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88e16948_5195_4cab_88ae_6a129e4a3214.slice/crio-39fbaf1d2169fc531883aaa0b66429926937c728909d231b95cf50b5b03c816b WatchSource:0}: Error finding container 39fbaf1d2169fc531883aaa0b66429926937c728909d231b95cf50b5b03c816b: Status 404 returned error can't find the container with id 39fbaf1d2169fc531883aaa0b66429926937c728909d231b95cf50b5b03c816b
Sep 18 22:21:57.379672 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.379654    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:21:57.709977 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:21:57.709945    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f" event={"ID":"88e16948-5195-4cab-88ae-6a129e4a3214","Type":"ContainerStarted","Data":"39fbaf1d2169fc531883aaa0b66429926937c728909d231b95cf50b5b03c816b"}
Sep 18 22:22:00.727118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:00.727078    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f" event={"ID":"88e16948-5195-4cab-88ae-6a129e4a3214","Type":"ContainerStarted","Data":"05ff94d3bda409dae6f874fa876edc927775f98e4bee256d22adab445f2ae642"}
Sep 18 22:22:00.745929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:00.745877    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/openshift-lws-operator-fd8ccff4c-7n77f" podStartSLOduration=1.96789632 podStartE2EDuration="4.74586059s" totalImagesPullingTime="2.77796427s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:00.74345189 +0000 UTC m=+312.925036437" watchObservedRunningTime="2026-09-18 22:22:00.74586059 +0000 UTC m=+312.927445138"
Sep 18 22:22:09.696346 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.696313    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"]
Sep 18 22:22:09.698607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.698584    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.701510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.701489    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-lws-operator\"/\"lws-manager-config\""
Sep 18 22:22:09.701632 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.701576    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"webhook-server-cert\""
Sep 18 22:22:09.702614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.702595    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"metrics-server-cert\""
Sep 18 22:22:09.702723 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.702618    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-lws-operator\"/\"lws-controller-manager-dockercfg-rrh9w\""
Sep 18 22:22:09.712894 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.712868    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"]
Sep 18 22:22:09.827697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.827659    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/672eccbc-27de-424e-b198-09a14f3d4538-manager-config\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.827882 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.827716    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.827882 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.827768    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqxbn\" (UniqueName: \"kubernetes.io/projected/672eccbc-27de-424e-b198-09a14f3d4538-kube-api-access-wqxbn\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.827882 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.827801    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-metrics-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.929191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.929149    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.929389 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.929207    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqxbn\" (UniqueName: \"kubernetes.io/projected/672eccbc-27de-424e-b198-09a14f3d4538-kube-api-access-wqxbn\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.929389 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.929255    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-metrics-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.929389 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.929295    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/672eccbc-27de-424e-b198-09a14f3d4538-manager-config\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.930160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.930133    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/672eccbc-27de-424e-b198-09a14f3d4538-manager-config\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.931696 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.931676    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.931771 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.931751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/672eccbc-27de-424e-b198-09a14f3d4538-metrics-cert\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:09.938831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:09.938808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqxbn\" (UniqueName: \"kubernetes.io/projected/672eccbc-27de-424e-b198-09a14f3d4538-kube-api-access-wqxbn\") pod \"lws-controller-manager-5bdd6569c9-d7hp2\" (UID: \"672eccbc-27de-424e-b198-09a14f3d4538\") " pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:10.008834 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:10.008744    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:10.280213 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:10.280139    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"]
Sep 18 22:22:10.284157 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:22:10.284131    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod672eccbc_27de_424e_b198_09a14f3d4538.slice/crio-ec3e090b2f944d5d60b4a865b88d5df21b19fac4e3d0f3931ccd5a43154e9512 WatchSource:0}: Error finding container ec3e090b2f944d5d60b4a865b88d5df21b19fac4e3d0f3931ccd5a43154e9512: Status 404 returned error can't find the container with id ec3e090b2f944d5d60b4a865b88d5df21b19fac4e3d0f3931ccd5a43154e9512
Sep 18 22:22:10.767076 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:10.767016    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2" event={"ID":"672eccbc-27de-424e-b198-09a14f3d4538","Type":"ContainerStarted","Data":"ec3e090b2f944d5d60b4a865b88d5df21b19fac4e3d0f3931ccd5a43154e9512"}
Sep 18 22:22:17.376180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.376142    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"]
Sep 18 22:22:17.378503 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.378487    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.381624 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.381599    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-6vdvw\""
Sep 18 22:22:17.381624 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.381611    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 18 22:22:17.381817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.381610    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 18 22:22:17.381817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.381650    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 18 22:22:17.381993 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.381978    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 18 22:22:17.384650 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.384620    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-manifests\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.384772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.384679    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.384772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.384728    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-tmp\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.384772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.384756    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.384946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.384800    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wfpdz\" (UniqueName: \"kubernetes.io/projected/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-kube-api-access-wfpdz\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.406931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.406900    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"]
Sep 18 22:22:17.485752 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.485711    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wfpdz\" (UniqueName: \"kubernetes.io/projected/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-kube-api-access-wfpdz\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.485931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.485780    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-manifests\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.485931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.485806    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.485931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.485858    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-tmp\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.485931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.485885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.486350 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.486321    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-manifests\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.488410 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.488382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-tmp\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.488523 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.488426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-apiservice-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.488523 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.488492    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-webhook-cert\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.494552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.494530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wfpdz\" (UniqueName: \"kubernetes.io/projected/01c8de87-3bd7-4f76-88aa-cd5126e73cb6-kube-api-access-wfpdz\") pod \"opendatahub-operator-controller-manager-65c85dbc8b-xc6nz\" (UID: \"01c8de87-3bd7-4f76-88aa-cd5126e73cb6\") " pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.689384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.689347    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:17.971397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:17.971370    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"]
Sep 18 22:22:17.973548 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:22:17.973523    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01c8de87_3bd7_4f76_88aa_cd5126e73cb6.slice/crio-66a361dc88c581ea57569cbbcb435bb5d8ff2e8232ca541be1764811a4a7fc07 WatchSource:0}: Error finding container 66a361dc88c581ea57569cbbcb435bb5d8ff2e8232ca541be1764811a4a7fc07: Status 404 returned error can't find the container with id 66a361dc88c581ea57569cbbcb435bb5d8ff2e8232ca541be1764811a4a7fc07
Sep 18 22:22:18.801939 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:18.801864    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz" event={"ID":"01c8de87-3bd7-4f76-88aa-cd5126e73cb6","Type":"ContainerStarted","Data":"66a361dc88c581ea57569cbbcb435bb5d8ff2e8232ca541be1764811a4a7fc07"}
Sep 18 22:22:21.817050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:21.817001    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz" event={"ID":"01c8de87-3bd7-4f76-88aa-cd5126e73cb6","Type":"ContainerStarted","Data":"9e27b52f16f2791da9b80ef13d846ead6b72453c850ac9f6955ba9545402d4c3"}
Sep 18 22:22:22.823418 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:22.823382    2551 generic.go:356] "Generic (PLEG): container finished" podID="01c8de87-3bd7-4f76-88aa-cd5126e73cb6" containerID="9e27b52f16f2791da9b80ef13d846ead6b72453c850ac9f6955ba9545402d4c3" exitCode=0
Sep 18 22:22:22.823810 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:22.823425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz" event={"ID":"01c8de87-3bd7-4f76-88aa-cd5126e73cb6","Type":"ContainerDied","Data":"9e27b52f16f2791da9b80ef13d846ead6b72453c850ac9f6955ba9545402d4c3"}
Sep 18 22:22:23.829710 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:23.829675    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz" event={"ID":"01c8de87-3bd7-4f76-88aa-cd5126e73cb6","Type":"ContainerStarted","Data":"a7be1ee6065671f55bca60594368e2e9157d2fd28fca45c72c8506b76ad073d4"}
Sep 18 22:22:23.830115 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:23.829775    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:23.856398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:23.856329    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz" podStartSLOduration=2.552232766 podStartE2EDuration="6.856307684s" totalImagesPullingTime="3.233848852s" totalInitContainerRuntime="1.070226066s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:23.85357909 +0000 UTC m=+336.035163636" watchObservedRunningTime="2026-09-18 22:22:23.856307684 +0000 UTC m=+336.037892230"
Sep 18 22:22:28.852809 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:28.852773    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2" event={"ID":"672eccbc-27de-424e-b198-09a14f3d4538","Type":"ContainerStarted","Data":"4d0ef9083bf9c31fb95f338c0a0010ed5eb098e434e76c5e38100dd27d99c9e8"}
Sep 18 22:22:28.853215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:28.852904    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:28.903873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:28.903814    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2" podStartSLOduration=2.30333001 podStartE2EDuration="19.903796708s" totalImagesPullingTime="17.600466698s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:09 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:28.902532947 +0000 UTC m=+341.084117492" watchObservedRunningTime="2026-09-18 22:22:28.903796708 +0000 UTC m=+341.085381253"
Sep 18 22:22:34.836994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:34.836967    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-65c85dbc8b-xc6nz"
Sep 18 22:22:37.244317 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.244284    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"]
Sep 18 22:22:37.258276 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.258243    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.265422 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.265392    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 18 22:22:37.265675 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.265543    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-lj82g\""
Sep 18 22:22:37.266731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.266352    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 18 22:22:37.266731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.266561    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"]
Sep 18 22:22:37.267296 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.267267    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 18 22:22:37.267846 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.267825    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 18 22:22:37.351290 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.351256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66432609-045a-46a7-84f8-3bbef484f2cc-tmp\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.351879 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.351846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksxvz\" (UniqueName: \"kubernetes.io/projected/66432609-045a-46a7-84f8-3bbef484f2cc-kube-api-access-ksxvz\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.351991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.351937    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.453409 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.453372    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.453581 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.453446    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66432609-045a-46a7-84f8-3bbef484f2cc-tmp\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.453581 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.453506    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksxvz\" (UniqueName: \"kubernetes.io/projected/66432609-045a-46a7-84f8-3bbef484f2cc-kube-api-access-ksxvz\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.453581 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:22:37.453536    2551 secret.go:189] Couldn't get secret openshift-ingress/kube-auth-proxy-tls: secret "kube-auth-proxy-tls" not found
Sep 18 22:22:37.453718 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:22:37.453613    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs podName:66432609-045a-46a7-84f8-3bbef484f2cc nodeName:}" failed. No retries permitted until 2026-09-18 22:22:37.953594748 +0000 UTC m=+350.135179291 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "tls-certs" (UniqueName: "kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs") pod "kube-auth-proxy-6b786b9564-bsppv" (UID: "66432609-045a-46a7-84f8-3bbef484f2cc") : secret "kube-auth-proxy-tls" not found
Sep 18 22:22:37.456117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.456089    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/66432609-045a-46a7-84f8-3bbef484f2cc-tmp\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.469244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.469219    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksxvz\" (UniqueName: \"kubernetes.io/projected/66432609-045a-46a7-84f8-3bbef484f2cc-kube-api-access-ksxvz\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.959761 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.959718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:37.962162 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:37.962141    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/66432609-045a-46a7-84f8-3bbef484f2cc-tls-certs\") pod \"kube-auth-proxy-6b786b9564-bsppv\" (UID: \"66432609-045a-46a7-84f8-3bbef484f2cc\") " pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:38.170224 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:38.170192    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"
Sep 18 22:22:38.470311 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:38.470283    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-6b786b9564-bsppv"]
Sep 18 22:22:38.472824 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:22:38.472798    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66432609_045a_46a7_84f8_3bbef484f2cc.slice/crio-a789b066788a67dfb83a2cfdc04f2a83ba630afd6ca6696d92346f683a78b949 WatchSource:0}: Error finding container a789b066788a67dfb83a2cfdc04f2a83ba630afd6ca6696d92346f683a78b949: Status 404 returned error can't find the container with id a789b066788a67dfb83a2cfdc04f2a83ba630afd6ca6696d92346f683a78b949
Sep 18 22:22:38.897701 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:38.897667    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv" event={"ID":"66432609-045a-46a7-84f8-3bbef484f2cc","Type":"ContainerStarted","Data":"a789b066788a67dfb83a2cfdc04f2a83ba630afd6ca6696d92346f683a78b949"}
Sep 18 22:22:39.860093 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:39.860015    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-lws-operator/lws-controller-manager-5bdd6569c9-d7hp2"
Sep 18 22:22:41.916711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:41.916673    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv" event={"ID":"66432609-045a-46a7-84f8-3bbef484f2cc","Type":"ContainerStarted","Data":"7c0bbe60c2cd03cf97b881ecc6ff61e1f59e97a98c4f98f71e9855db58675c5f"}
Sep 18 22:22:41.933909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:22:41.933856    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-6b786b9564-bsppv" podStartSLOduration=2.299396368 podStartE2EDuration="4.933826333s" totalImagesPullingTime="2.634429965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:22:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:22:41.932769249 +0000 UTC m=+354.114353796" watchObservedRunningTime="2026-09-18 22:22:41.933826333 +0000 UTC m=+354.115410878"
Sep 18 22:24:13.965737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.965654    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"]
Sep 18 22:24:13.968919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.968902    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:13.971702 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.971674    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 18 22:24:13.971702 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.971685    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 18 22:24:13.972148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.972125    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 18 22:24:13.972270 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.972146    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-kl9kq\""
Sep 18 22:24:13.984390 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.984366    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"]
Sep 18 22:24:13.992104 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:13.992079    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vt727\" (UniqueName: \"kubernetes.io/projected/28893176-b880-4223-addf-fdced418f439-kube-api-access-vt727\") pod \"dns-operator-controller-manager-7b84cd7c9c-rjkmb\" (UID: \"28893176-b880-4223-addf-fdced418f439\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:14.092815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:14.092777    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vt727\" (UniqueName: \"kubernetes.io/projected/28893176-b880-4223-addf-fdced418f439-kube-api-access-vt727\") pod \"dns-operator-controller-manager-7b84cd7c9c-rjkmb\" (UID: \"28893176-b880-4223-addf-fdced418f439\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:14.103331 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:14.103302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vt727\" (UniqueName: \"kubernetes.io/projected/28893176-b880-4223-addf-fdced418f439-kube-api-access-vt727\") pod \"dns-operator-controller-manager-7b84cd7c9c-rjkmb\" (UID: \"28893176-b880-4223-addf-fdced418f439\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:14.283536 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:14.283441    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:14.557815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:14.557749    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"]
Sep 18 22:24:14.560138 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:24:14.560108    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod28893176_b880_4223_addf_fdced418f439.slice/crio-b328cec30c5910d0ba5ddb7c358d88cd9d2f3097cdc2fa58bf4cc868545cef6a WatchSource:0}: Error finding container b328cec30c5910d0ba5ddb7c358d88cd9d2f3097cdc2fa58bf4cc868545cef6a: Status 404 returned error can't find the container with id b328cec30c5910d0ba5ddb7c358d88cd9d2f3097cdc2fa58bf4cc868545cef6a
Sep 18 22:24:15.324300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:15.324258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb" event={"ID":"28893176-b880-4223-addf-fdced418f439","Type":"ContainerStarted","Data":"b328cec30c5910d0ba5ddb7c358d88cd9d2f3097cdc2fa58bf4cc868545cef6a"}
Sep 18 22:24:17.336962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:17.336921    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb" event={"ID":"28893176-b880-4223-addf-fdced418f439","Type":"ContainerStarted","Data":"803ae9c186a82e08479f4a254c2a18d007498901ee2887fb8b2927bddb7c9fde"}
Sep 18 22:24:17.337385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:17.337025    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:17.358663 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:17.358610    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb" podStartSLOduration=1.6815091789999999 podStartE2EDuration="4.358592349s" totalImagesPullingTime="2.67708317s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:24:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:24:17.35639079 +0000 UTC m=+449.537975337" watchObservedRunningTime="2026-09-18 22:24:17.358592349 +0000 UTC m=+449.540176896"
Sep 18 22:24:28.345096 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:28.345064    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-rjkmb"
Sep 18 22:24:29.799417 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.799380    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:24:29.806537 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.806510    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:29.809266 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.809243    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v2hc2\""
Sep 18 22:24:29.828567 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.828542    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:24:29.931107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.931068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:29.931293 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:29.931148    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbmpc\" (UniqueName: \"kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.032499 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.032463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.032499 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.032506    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbmpc\" (UniqueName: \"kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.032895 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.032871    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.046594 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.046568    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbmpc\" (UniqueName: \"kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9wrj6\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.117097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.117009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:30.400350 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:30.400327    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:24:30.402904 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:24:30.402873    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18261b97_897d_4a5d_b471_75099643a20f.slice/crio-45718b8e6e8bd7471da074683114973abf9576d1975fede3646d2029fca06d30 WatchSource:0}: Error finding container 45718b8e6e8bd7471da074683114973abf9576d1975fede3646d2029fca06d30: Status 404 returned error can't find the container with id 45718b8e6e8bd7471da074683114973abf9576d1975fede3646d2029fca06d30
Sep 18 22:24:31.397944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:31.397901    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerStarted","Data":"45718b8e6e8bd7471da074683114973abf9576d1975fede3646d2029fca06d30"}
Sep 18 22:24:34.416434 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:34.416400    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerStarted","Data":"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"}
Sep 18 22:24:34.416832 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:34.416489    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:24:34.443600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:34.443550    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" podStartSLOduration=1.524277611 podStartE2EDuration="5.443535732s" totalImagesPullingTime="3.919258121s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:24:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:24:34.441373333 +0000 UTC m=+466.622957878" watchObservedRunningTime="2026-09-18 22:24:34.443535732 +0000 UTC m=+466.625120275"
Sep 18 22:24:45.425321 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:24:45.425288    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:25:04.185222 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.185187    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:25:04.189946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.189923    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.192265 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.192242    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-rsg24\""
Sep 18 22:25:04.192718 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.192693    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Sep 18 22:25:04.197663 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.197597    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:25:04.227676 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.227636    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zcgkx\" (UniqueName: \"kubernetes.io/projected/cc33c016-d525-4979-bfb3-9f226deb133d-kube-api-access-zcgkx\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.227833 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.227697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc33c016-d525-4979-bfb3-9f226deb133d-config-file\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.228867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.228843    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:25:04.328730 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.328694    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zcgkx\" (UniqueName: \"kubernetes.io/projected/cc33c016-d525-4979-bfb3-9f226deb133d-kube-api-access-zcgkx\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.328905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.328745    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc33c016-d525-4979-bfb3-9f226deb133d-config-file\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.329427 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.329409    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/cc33c016-d525-4979-bfb3-9f226deb133d-config-file\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.339044 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.338997    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zcgkx\" (UniqueName: \"kubernetes.io/projected/cc33c016-d525-4979-bfb3-9f226deb133d-kube-api-access-zcgkx\") pod \"limitador-limitador-84994f7ffc-6sllq\" (UID: \"cc33c016-d525-4979-bfb3-9f226deb133d\") " pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.501960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.501840    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:04.774701 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:04.774670    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:25:04.777390 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:25:04.777367    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc33c016_d525_4979_bfb3_9f226deb133d.slice/crio-4c16e7bbd48232358a216343005896ce10f4df66800e042f94e10143c29ac0b1 WatchSource:0}: Error finding container 4c16e7bbd48232358a216343005896ce10f4df66800e042f94e10143c29ac0b1: Status 404 returned error can't find the container with id 4c16e7bbd48232358a216343005896ce10f4df66800e042f94e10143c29ac0b1
Sep 18 22:25:05.572048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:05.572002    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq" event={"ID":"cc33c016-d525-4979-bfb3-9f226deb133d","Type":"ContainerStarted","Data":"4c16e7bbd48232358a216343005896ce10f4df66800e042f94e10143c29ac0b1"}
Sep 18 22:25:06.578492 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:06.578460    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq" event={"ID":"cc33c016-d525-4979-bfb3-9f226deb133d","Type":"ContainerStarted","Data":"38eddffba1f8f5aeab7df447b462f6c4f471b8723a04ec12b58ca1224e246cf1"}
Sep 18 22:25:06.578942 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:06.578580    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:25:06.601647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:06.601593    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq" podStartSLOduration=1.513224085 podStartE2EDuration="2.601575778s" totalImagesPullingTime="1.088351693s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:25:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:25:06.599826659 +0000 UTC m=+498.781411198" watchObservedRunningTime="2026-09-18 22:25:06.601575778 +0000 UTC m=+498.783160324"
Sep 18 22:25:17.584870 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:25:17.584836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-84994f7ffc-6sllq"
Sep 18 22:26:15.980097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:26:15.980055    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:26:48.239636 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:26:48.239605    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:26:48.240233 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:26:48.239778    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:27:08.024089 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:08.024052    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:25.195855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:25.195767    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:29.569705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:29.569661    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:32.181129 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.181090    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"]
Sep 18 22:27:32.185086 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.185067    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.187637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.187616    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-s9rh9\""
Sep 18 22:27:32.188763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.188746    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 18 22:27:32.188763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.188751    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 18 22:27:32.188928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.188793    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Sep 18 22:27:32.194733 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.194708    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"]
Sep 18 22:27:32.360416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360388    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jdh5p\" (UniqueName: \"kubernetes.io/projected/0c00fc26-966d-4397-8709-83f94c93f7d2-kube-api-access-jdh5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.360558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360436    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.360558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360498    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00fc26-966d-4397-8709-83f94c93f7d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.360558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360517    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.360558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360547    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.360694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.360564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461417    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461463    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00fc26-966d-4397-8709-83f94c93f7d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461524    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461549    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jdh5p\" (UniqueName: \"kubernetes.io/projected/0c00fc26-966d-4397-8709-83f94c93f7d2-kube-api-access-jdh5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-model-cache\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.461940 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461921    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-home\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.462002 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.461929    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.463981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.463954    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c00fc26-966d-4397-8709-83f94c93f7d2-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.464126 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.464081    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c00fc26-966d-4397-8709-83f94c93f7d2-dshm\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.469081 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.469060    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jdh5p\" (UniqueName: \"kubernetes.io/projected/0c00fc26-966d-4397-8709-83f94c93f7d2-kube-api-access-jdh5p\") pod \"e2e-trlp-test-simulated-kserve-598466548d-sh8gb\" (UID: \"0c00fc26-966d-4397-8709-83f94c93f7d2\") " pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.495681 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.495659    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:32.766845 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.766822    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"]
Sep 18 22:27:32.769221 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:27:32.769181    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0c00fc26_966d_4397_8709_83f94c93f7d2.slice/crio-c10de93ebb49338442e6b8aaf53d25f86041ebe46b922ddfb95ba1ebe6bcf71a WatchSource:0}: Error finding container c10de93ebb49338442e6b8aaf53d25f86041ebe46b922ddfb95ba1ebe6bcf71a: Status 404 returned error can't find the container with id c10de93ebb49338442e6b8aaf53d25f86041ebe46b922ddfb95ba1ebe6bcf71a
Sep 18 22:27:32.770580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:32.770561    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:27:33.255891 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:33.255835    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb" event={"ID":"0c00fc26-966d-4397-8709-83f94c93f7d2","Type":"ContainerStarted","Data":"c10de93ebb49338442e6b8aaf53d25f86041ebe46b922ddfb95ba1ebe6bcf71a"}
Sep 18 22:27:33.890803 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:33.890774    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:34.262142 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:34.262100    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb" event={"ID":"0c00fc26-966d-4397-8709-83f94c93f7d2","Type":"ContainerStarted","Data":"feb8f7c2a8cd6e5a56e4d891b19768a47db90507d6e4eac3b60a9fdf93c2c25b"}
Sep 18 22:27:34.286350 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:34.286300    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb" podStartSLOduration=1.29001666 podStartE2EDuration="2.286284794s" totalImagesPullingTime="996.268134ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:27:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:27:34.284933833 +0000 UTC m=+646.466518378" watchObservedRunningTime="2026-09-18 22:27:34.286284794 +0000 UTC m=+646.467869383"
Sep 18 22:27:40.912258 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:40.912212    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:42.495859 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:42.495827    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:42.496363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:42.495897    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:42.512078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:42.512045    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:43.314265 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:43.314230    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-598466548d-sh8gb"
Sep 18 22:27:47.112464 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:47.112423    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:27:51.135757 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.135722    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"]
Sep 18 22:27:51.139203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.139183    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.142150 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.142123    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"premium-simulated-simulated-premium-kserve-self-signed-certs\""
Sep 18 22:27:51.154932 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.154909    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"]
Sep 18 22:27:51.224772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224734    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zrq8r\" (UniqueName: \"kubernetes.io/projected/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-kube-api-access-zrq8r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.224920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224780    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.224920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224806    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.224920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224878    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.225075 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224923    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.225075 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.224967    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325514    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zrq8r\" (UniqueName: \"kubernetes.io/projected/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-kube-api-access-zrq8r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325596    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325644    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325672    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.325973 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.325922    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.326116 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.326013    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tmp-dir\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.326116 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.326074    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-model-cache\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.326231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.326147    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-home\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.327945 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.327919    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-dshm\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.328211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.328185    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-tls-certs\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.334162 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.334136    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zrq8r\" (UniqueName: \"kubernetes.io/projected/a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e-kube-api-access-zrq8r\") pod \"premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5\" (UID: \"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e\") " pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.448583 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.448540    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:27:51.723597 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:51.723572    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"]
Sep 18 22:27:51.725202 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:27:51.725174    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poda0e99161_4fe4_45a0_992f_d4dc0b1d9d7e.slice/crio-cfdf64535158f2dddff892b1724dc99def7ad0cf49328119c4c8879b0a564ad7 WatchSource:0}: Error finding container cfdf64535158f2dddff892b1724dc99def7ad0cf49328119c4c8879b0a564ad7: Status 404 returned error can't find the container with id cfdf64535158f2dddff892b1724dc99def7ad0cf49328119c4c8879b0a564ad7
Sep 18 22:27:52.347474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:52.347428    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5" event={"ID":"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e","Type":"ContainerStarted","Data":"bee29384e0c8723cd218a83bd526d8196c8f75d50b8df0d23b1f4b8232e754ce"}
Sep 18 22:27:52.347474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:52.347474    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5" event={"ID":"a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e","Type":"ContainerStarted","Data":"cfdf64535158f2dddff892b1724dc99def7ad0cf49328119c4c8879b0a564ad7"}
Sep 18 22:27:52.366939 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:52.366882    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5" podStartSLOduration=1.178427561 podStartE2EDuration="1.366866503s" totalImagesPullingTime="188.438942ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:27:51 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:27:52.365025516 +0000 UTC m=+664.546610080" watchObservedRunningTime="2026-09-18 22:27:52.366866503 +0000 UTC m=+664.548451049"
Sep 18 22:27:52.994235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:27:52.994199    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:28:01.449136 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:28:01.449100    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:28:01.449705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:28:01.449152    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:28:01.461674 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:28:01.461649    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:28:02.405190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:28:02.405160    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5"
Sep 18 22:29:08.312761 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.312727    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-618f5f"]
Sep 18 22:29:08.316193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.316175    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:08.319442 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.319421    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lgx4p\""
Sep 18 22:29:08.336409 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.336385    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-618f5f"]
Sep 18 22:29:08.419937 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.419895    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nxb9l\" (UniqueName: \"kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l\") pod \"test-curl-7553-618f5f\" (UID: \"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5\") " pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:08.520998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.520961    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nxb9l\" (UniqueName: \"kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l\") pod \"test-curl-7553-618f5f\" (UID: \"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5\") " pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:08.548642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.548613    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nxb9l\" (UniqueName: \"kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l\") pod \"test-curl-7553-618f5f\" (UID: \"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5\") " pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:08.625330 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.625246    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:08.894236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:08.894168    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-618f5f"]
Sep 18 22:29:08.897328 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:08.897292    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8727b0d6_713f_47f5_a6fd_905cf6fbf3e5.slice/crio-4a1a53cd4348bd4404977e741bbcfc7ad5a4e5c98f6e69d9586aab2d2bcc4b31 WatchSource:0}: Error finding container 4a1a53cd4348bd4404977e741bbcfc7ad5a4e5c98f6e69d9586aab2d2bcc4b31: Status 404 returned error can't find the container with id 4a1a53cd4348bd4404977e741bbcfc7ad5a4e5c98f6e69d9586aab2d2bcc4b31
Sep 18 22:29:09.716152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:09.716116    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-618f5f" event={"ID":"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5","Type":"ContainerStarted","Data":"4a1a53cd4348bd4404977e741bbcfc7ad5a4e5c98f6e69d9586aab2d2bcc4b31"}
Sep 18 22:29:10.354235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:10.354189    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:29:11.726766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:11.726728    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-618f5f" event={"ID":"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5","Type":"ContainerStarted","Data":"dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67"}
Sep 18 22:29:11.742642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:11.742583    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-618f5f" podStartSLOduration=1.74280575 podStartE2EDuration="3.742566314s" totalImagesPullingTime="1.999760564s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:08 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:11.741778353 +0000 UTC m=+743.923362898" watchObservedRunningTime="2026-09-18 22:29:11.742566314 +0000 UTC m=+743.924150861"
Sep 18 22:29:12.360475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.360432    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-618f5f"]
Sep 18 22:29:12.362899 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.362867    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-618f5f"]
Sep 18 22:29:12.481069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.481022    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7553-3b7f00"]
Sep 18 22:29:12.481414 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.481400    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" containerName="test-curl-7553-618f5f"
Sep 18 22:29:12.481456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.481417    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" containerName="test-curl-7553-618f5f"
Sep 18 22:29:12.481504 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.481493    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" containerName="test-curl-7553-618f5f"
Sep 18 22:29:12.491378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.491351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-3b7f00"]
Sep 18 22:29:12.491522 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.491445    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:12.594463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.594431    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:29:12.597364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.597347    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.600991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.600970    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 22:29:12.601410 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.601392    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w4-f9cf33-serving-cert\""
Sep 18 22:29:12.601799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.601780    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xxrx\""
Sep 18 22:29:12.601875 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.601782    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:29:12.601926 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.601892    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 22:29:12.610165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.610139    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:29:12.655417 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.655383    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nsbfh\" (UniqueName: \"kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh\") pod \"test-curl-7553-3b7f00\" (UID: \"6693b9b4-0682-4d75-ae07-afdedfc7bb90\") " pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:12.732516 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:12.732425    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-618f5f" container="test-curl-7553-618f5f"
Sep 18 22:29:12.756193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.756159    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nsbfh\" (UniqueName: \"kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh\") pod \"test-curl-7553-3b7f00\" (UID: \"6693b9b4-0682-4d75-ae07-afdedfc7bb90\") " pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:12.756364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.756204    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.756364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.756322    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzqb5\" (UniqueName: \"kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.756435 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.756396    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.756486 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.756442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.769402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.769373    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nsbfh\" (UniqueName: \"kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh\") pod \"test-curl-7553-3b7f00\" (UID: \"6693b9b4-0682-4d75-ae07-afdedfc7bb90\") " pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:12.801055 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.801005    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:12.857965 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.857926    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.858099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.857990    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzqb5\" (UniqueName: \"kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.858099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.858056    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.858218 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.858102    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.858883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.858856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.859367 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:12.858983    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-serving-cert: secret "maas-api-e2e-worker-w4-f9cf33-serving-cert" not found
Sep 18 22:29:12.859367 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:12.859079    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls podName:7ad8cf37-41e1-4b82-b415-ed0b6c49e54b nodeName:}" failed. No retries permitted until 2026-09-18 22:29:13.359058196 +0000 UTC m=+745.540642735 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls") pod "maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" (UID: "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b") : secret "maas-api-e2e-worker-w4-f9cf33-serving-cert" not found
Sep 18 22:29:12.862575 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.862531    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:12.870050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:12.869983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzqb5\" (UniqueName: \"kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:13.076300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.076213    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7553-3b7f00"]
Sep 18 22:29:13.079377 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:13.079349    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6693b9b4_0682_4d75_ae07_afdedfc7bb90.slice/crio-cde4012d867e717a19c23b9cad25483455e6270f00c06330466308078ffd8fad WatchSource:0}: Error finding container cde4012d867e717a19c23b9cad25483455e6270f00c06330466308078ffd8fad: Status 404 returned error can't find the container with id cde4012d867e717a19c23b9cad25483455e6270f00c06330466308078ffd8fad
Sep 18 22:29:13.362831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.362747    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:13.365126 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.365106    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") pod \"maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:13.507189 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.507149    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:13.738117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.738076    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-3b7f00" event={"ID":"6693b9b4-0682-4d75-ae07-afdedfc7bb90","Type":"ContainerStarted","Data":"cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2"}
Sep 18 22:29:13.738549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.738126    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7553-3b7f00" event={"ID":"6693b9b4-0682-4d75-ae07-afdedfc7bb90","Type":"ContainerStarted","Data":"cde4012d867e717a19c23b9cad25483455e6270f00c06330466308078ffd8fad"}
Sep 18 22:29:13.738549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.738215    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-618f5f" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" containerName="test-curl-7553-618f5f" containerID="cri-o://dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67" gracePeriod=2
Sep 18 22:29:13.755971 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:13.755937    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-618f5f\" is forbidden: User \"system:node:ip-10-0-129-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-37.ec2.internal' and this object" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:13.756121 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.755984    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7553-3b7f00" podStartSLOduration=1.755970225 podStartE2EDuration="1.755970225s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:13.753922633 +0000 UTC m=+745.935507178" watchObservedRunningTime="2026-09-18 22:29:13.755970225 +0000 UTC m=+745.937554799"
Sep 18 22:29:13.795517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:13.795486    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:29:13.796778 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:13.796749    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7ad8cf37_41e1_4b82_b415_ed0b6c49e54b.slice/crio-56cabe246fe0ad25c51a04b763fe51d27a38f9c72918b05fa31fb071ebced12f WatchSource:0}: Error finding container 56cabe246fe0ad25c51a04b763fe51d27a38f9c72918b05fa31fb071ebced12f: Status 404 returned error can't find the container with id 56cabe246fe0ad25c51a04b763fe51d27a38f9c72918b05fa31fb071ebced12f
Sep 18 22:29:14.380950 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.380918    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7553-3b7f00"]
Sep 18 22:29:14.387696 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.385839    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7553-3b7f00"]
Sep 18 22:29:14.521540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.521506    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:29:14.522111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.522088    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" containerName="test-curl-7553-3b7f00"
Sep 18 22:29:14.522111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.522114    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" containerName="test-curl-7553-3b7f00"
Sep 18 22:29:14.522298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.522219    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" containerName="test-curl-7553-3b7f00"
Sep 18 22:29:14.525615 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.525590    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.529000 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.528585    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w0-c5fdd3-dockercfg-n7wcf\""
Sep 18 22:29:14.529000 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.528624    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w0-c5fdd3\""
Sep 18 22:29:14.542204 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.542175    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:29:14.572223 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.572185    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.572411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.572360    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lh82\" (UniqueName: \"kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.673299 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.673266    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lh82\" (UniqueName: \"kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.673537 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.673325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.674160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.674138    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.682586 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.682557    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lh82\" (UniqueName: \"kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82\") pod \"payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") " pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:14.744930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.744891    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" event={"ID":"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b","Type":"ContainerStarted","Data":"56cabe246fe0ad25c51a04b763fe51d27a38f9c72918b05fa31fb071ebced12f"}
Sep 18 22:29:14.745352 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:14.745098    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7553-3b7f00" container="test-curl-7553-3b7f00"
Sep 18 22:29:14.841384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:14.841350    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:15.140052 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:15.139898    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:29:15.614951 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:15.614910    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4e091709_cf96_42ed_8e91_43bd483c8f62.slice/crio-3ae7ebe39424afc719c272cc6709e90c88bfa253302ae122655fa3da759e6909 WatchSource:0}: Error finding container 3ae7ebe39424afc719c272cc6709e90c88bfa253302ae122655fa3da759e6909: Status 404 returned error can't find the container with id 3ae7ebe39424afc719c272cc6709e90c88bfa253302ae122655fa3da759e6909
Sep 18 22:29:15.751907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:15.751873    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" event={"ID":"4e091709-cf96-42ed-8e91-43bd483c8f62","Type":"ContainerStarted","Data":"3ae7ebe39424afc719c272cc6709e90c88bfa253302ae122655fa3da759e6909"}
Sep 18 22:29:15.752275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:15.751942    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7553-3b7f00" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" containerName="test-curl-7553-3b7f00" containerID="cri-o://cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2" gracePeriod=2
Sep 18 22:29:15.754347 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:15.754317    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-3b7f00\" is forbidden: User \"system:node:ip-10-0-129-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-37.ec2.internal' and this object" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:16.165802 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.165771    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:16.168331 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:16.168287    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7553-3b7f00\" is forbidden: User \"system:node:ip-10-0-129-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-37.ec2.internal' and this object" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:16.186343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.186316    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nxb9l\" (UniqueName: \"kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l\") pod \"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5\" (UID: \"8727b0d6-713f-47f5-a6fd-905cf6fbf3e5\") "
Sep 18 22:29:16.188528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.188500    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l" (OuterVolumeSpecName: "kube-api-access-nxb9l") pod "8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" (UID: "8727b0d6-713f-47f5-a6fd-905cf6fbf3e5"). InnerVolumeSpecName "kube-api-access-nxb9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:16.287138 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.287049    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nxb9l\" (UniqueName: \"kubernetes.io/projected/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5-kube-api-access-nxb9l\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:16.295873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.295837    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" path="/var/lib/kubelet/pods/8727b0d6-713f-47f5-a6fd-905cf6fbf3e5/volumes"
Sep 18 22:29:16.469744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.469609    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:29:16.473461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.473433    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.479641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.479616    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-519912e2-openshift-default-dockercfg-rwhv6\""
Sep 18 22:29:16.479871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.479856    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 18 22:29:16.489737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.489402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.489737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.489690    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.489912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.489760    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.489912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.489828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.490171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.490077    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.490171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.490105    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.490171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.490128    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wgcfg\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.490378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.490232    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.490378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.490277    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.496939 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.496875    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:29:16.591220 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591220 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591181    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591227    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591293    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591314    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591337    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wgcfg\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591389    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.591722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.591458    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.592156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.592092    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.592335 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.592308    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.592514 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.592453    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.592514 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.592478    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.592702 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.592684    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.594500 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.594477    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.595452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.595430    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.603756 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.603731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wgcfg\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.603848 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.603763    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token\") pod \"e2e-ait-519912e2-openshift-default-866d855bc6-27sg2\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") " pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:16.758400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.758362    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" event={"ID":"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b","Type":"ContainerStarted","Data":"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"}
Sep 18 22:29:16.758846 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.758453    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:16.759734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.759708    2551 generic.go:356] "Generic (PLEG): container finished" podID="8727b0d6-713f-47f5-a6fd-905cf6fbf3e5" containerID="dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67" exitCode=137
Sep 18 22:29:16.759873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.759746    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-618f5f"
Sep 18 22:29:16.759873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.759812    2551 scope.go:117] "RemoveContainer" containerID="dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67"
Sep 18 22:29:16.770109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.770087    2551 scope.go:117] "RemoveContainer" containerID="dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67"
Sep 18 22:29:16.770522 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:16.770499    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67\": container with ID starting with dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67 not found: ID does not exist" containerID="dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67"
Sep 18 22:29:16.770607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.770528    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67"} err="failed to get container status \"dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67\": rpc error: code = NotFound desc = could not find container \"dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67\": container with ID starting with dc517f69d6f81925b1e53a7cc636d255fdd35d4c8f7146c2eb522b04cf0ebd67 not found: ID does not exist"
Sep 18 22:29:16.781995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.781948    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" podStartSLOduration=2.502559935 podStartE2EDuration="4.781932407s" totalImagesPullingTime="2.279372472s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:16.779932328 +0000 UTC m=+748.961516872" watchObservedRunningTime="2026-09-18 22:29:16.781932407 +0000 UTC m=+748.963516969"
Sep 18 22:29:16.787550 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:16.787503    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:17.084983 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:17.084952    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:29:18.097272 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:18.097204    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod113471d7_5659_42df_9fcc_236e4b38cc20.slice/crio-d1cf68de137483033650ca6138f293c481c5cace19f84f6ac960954652594458 WatchSource:0}: Error finding container d1cf68de137483033650ca6138f293c481c5cace19f84f6ac960954652594458: Status 404 returned error can't find the container with id d1cf68de137483033650ca6138f293c481c5cace19f84f6ac960954652594458
Sep 18 22:29:18.224276 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.224247    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:18.307236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.307207    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nsbfh\" (UniqueName: \"kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh\") pod \"6693b9b4-0682-4d75-ae07-afdedfc7bb90\" (UID: \"6693b9b4-0682-4d75-ae07-afdedfc7bb90\") "
Sep 18 22:29:18.309315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.309289    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh" (OuterVolumeSpecName: "kube-api-access-nsbfh") pod "6693b9b4-0682-4d75-ae07-afdedfc7bb90" (UID: "6693b9b4-0682-4d75-ae07-afdedfc7bb90"). InnerVolumeSpecName "kube-api-access-nsbfh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:18.408320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.408286    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nsbfh\" (UniqueName: \"kubernetes.io/projected/6693b9b4-0682-4d75-ae07-afdedfc7bb90-kube-api-access-nsbfh\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:18.773059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.772938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" event={"ID":"4e091709-cf96-42ed-8e91-43bd483c8f62","Type":"ContainerStarted","Data":"07747f3e5caf1201ca1f958f68415f69a5e6ebe807c2c912043b9cfc2b577726"}
Sep 18 22:29:18.773236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.773127    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:18.774347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.774321    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerStarted","Data":"d1cf68de137483033650ca6138f293c481c5cace19f84f6ac960954652594458"}
Sep 18 22:29:18.775462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.775437    2551 generic.go:356] "Generic (PLEG): container finished" podID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" containerID="cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2" exitCode=137
Sep 18 22:29:18.775533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.775483    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7553-3b7f00"
Sep 18 22:29:18.775533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.775499    2551 scope.go:117] "RemoveContainer" containerID="cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2"
Sep 18 22:29:18.784247 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.784229    2551 scope.go:117] "RemoveContainer" containerID="cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2"
Sep 18 22:29:18.784541 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:18.784519    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2\": container with ID starting with cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2 not found: ID does not exist" containerID="cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2"
Sep 18 22:29:18.784581 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.784558    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2"} err="failed to get container status \"cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2\": rpc error: code = NotFound desc = could not find container \"cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2\": container with ID starting with cc2d7d7fe317115ef284a5fde1b53e1e00e28e472dc872e9f2a324ee88dba3b2 not found: ID does not exist"
Sep 18 22:29:18.790669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:18.790632    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" podStartSLOduration=2.259679076 podStartE2EDuration="4.79062011s" totalImagesPullingTime="2.530941034s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:14 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:18.790040205 +0000 UTC m=+750.971624741" watchObservedRunningTime="2026-09-18 22:29:18.79062011 +0000 UTC m=+750.972204655"
Sep 18 22:29:20.291821 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:20.291785    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6693b9b4-0682-4d75-ae07-afdedfc7bb90" path="/var/lib/kubelet/pods/6693b9b4-0682-4d75-ae07-afdedfc7bb90/volumes"
Sep 18 22:29:20.819999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:20.819959    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:20.820109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:20.820061    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:20.820109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:20.820094    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:21.794920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:21.794887    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerStarted","Data":"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"}
Sep 18 22:29:21.818475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:21.818425    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podStartSLOduration=3.09890057 podStartE2EDuration="5.818410535s" totalImagesPullingTime="2.719509965s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:21.81750451 +0000 UTC m=+753.999089057" watchObservedRunningTime="2026-09-18 22:29:21.818410535 +0000 UTC m=+753.999995080"
Sep 18 22:29:22.771886 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:22.771858    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:29:22.788098 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:22.788068    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:22.793485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:22.793452    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:22.800268 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:22.800239    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:22.801260 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:22.801234    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:25.693580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.693545    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:25.696802 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.696783    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.699691 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.699670    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-519912e2-serving-cert\""
Sep 18 22:29:25.722050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.721997    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:25.779540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.779505    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.779680 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.779556    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-drbwf\" (UniqueName: \"kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.779680 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.779628    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.779680 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.779655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.880746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.880703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.880946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.880758    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-drbwf\" (UniqueName: \"kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.880946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.880821    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.880946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.880846    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.881547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.881523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.883303 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.883281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.883382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.883353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:25.891139 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:25.891114    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-drbwf\" (UniqueName: \"kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf\") pod \"maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:26.007560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.007464    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:26.284599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.284518    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:26.288728 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:26.288692    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c0003b2_4781_4799_836c_3e17fa243907.slice/crio-a746efdd2cede39e0a6139ebf8ded0bca379195627df32a749ee01b159d81e48 WatchSource:0}: Error finding container a746efdd2cede39e0a6139ebf8ded0bca379195627df32a749ee01b159d81e48: Status 404 returned error can't find the container with id a746efdd2cede39e0a6139ebf8ded0bca379195627df32a749ee01b159d81e48
Sep 18 22:29:26.324854 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.324820    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:29:26.485128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.485094    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:26.490243 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.490221    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.493444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.493420    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-519912e2\""
Sep 18 22:29:26.493444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.493431    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-519912e2-dockercfg-8d25r\""
Sep 18 22:29:26.512704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.512675    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:26.573915 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.573830    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:26.577445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.577424    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.588128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.588104    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sbvxd\" (UniqueName: \"kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.588241 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.588138    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.591636 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.591611    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:26.689427 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.689391    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.689571 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.689443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sbvxd\" (UniqueName: \"kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.689571 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.689502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.689571 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.689548    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cf77f\" (UniqueName: \"kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.690194 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.690176    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.698548 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.698530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sbvxd\" (UniqueName: \"kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd\") pod \"payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") " pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.790469 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.790435    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cf77f\" (UniqueName: \"kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.790614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.790517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.791170 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.791152    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.801348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.801322    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:26.804778 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.804755    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cf77f\" (UniqueName: \"kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f\") pod \"payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:26.821474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.821439    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" event={"ID":"5c0003b2-4781-4799-836c-3e17fa243907","Type":"ContainerStarted","Data":"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"}
Sep 18 22:29:26.821587 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.821507    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" event={"ID":"5c0003b2-4781-4799-836c-3e17fa243907","Type":"ContainerStarted","Data":"a746efdd2cede39e0a6139ebf8ded0bca379195627df32a749ee01b159d81e48"}
Sep 18 22:29:26.821587 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.821539    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:26.863975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.863925    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" podStartSLOduration=1.447922221 podStartE2EDuration="1.863909305s" totalImagesPullingTime="415.987084ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:25 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:26.862203269 +0000 UTC m=+759.043787827" watchObservedRunningTime="2026-09-18 22:29:26.863909305 +0000 UTC m=+759.045493850"
Sep 18 22:29:26.887312 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:26.887273    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:27.092286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.092109    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:27.094507 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:27.094481    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25097b86_a8d4_4a5f_af53_489a937bf956.slice/crio-3f8f168293cd140a53951ffcb45535babc5a395d96599b78c25380e21b847390 WatchSource:0}: Error finding container 3f8f168293cd140a53951ffcb45535babc5a395d96599b78c25380e21b847390: Status 404 returned error can't find the container with id 3f8f168293cd140a53951ffcb45535babc5a395d96599b78c25380e21b847390
Sep 18 22:29:27.177107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.177079    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:27.178421 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:27.178399    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod250cfe09_d331_449e_90a2_1ffe01e48022.slice/crio-abaf3078e910acbfe1af7bc07da9b1bea13de056c6ca89384d5a1a594a6cb092 WatchSource:0}: Error finding container abaf3078e910acbfe1af7bc07da9b1bea13de056c6ca89384d5a1a594a6cb092: Status 404 returned error can't find the container with id abaf3078e910acbfe1af7bc07da9b1bea13de056c6ca89384d5a1a594a6cb092
Sep 18 22:29:27.828395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.828354    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" event={"ID":"250cfe09-d331-449e-90a2-1ffe01e48022","Type":"ContainerStarted","Data":"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"}
Sep 18 22:29:27.828873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.828421    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:27.828873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.828448    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" event={"ID":"250cfe09-d331-449e-90a2-1ffe01e48022","Type":"ContainerStarted","Data":"abaf3078e910acbfe1af7bc07da9b1bea13de056c6ca89384d5a1a594a6cb092"}
Sep 18 22:29:27.829829 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.829806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" event={"ID":"25097b86-a8d4-4a5f-af53-489a937bf956","Type":"ContainerStarted","Data":"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"}
Sep 18 22:29:27.829901 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.829848    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" event={"ID":"25097b86-a8d4-4a5f-af53-489a937bf956","Type":"ContainerStarted","Data":"3f8f168293cd140a53951ffcb45535babc5a395d96599b78c25380e21b847390"}
Sep 18 22:29:27.846951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.846898    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" podStartSLOduration=1.846881386 podStartE2EDuration="1.846881386s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:27.845868164 +0000 UTC m=+760.027452710" watchObservedRunningTime="2026-09-18 22:29:27.846881386 +0000 UTC m=+760.028465929"
Sep 18 22:29:27.876685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:27.876615    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" podStartSLOduration=1.876593628 podStartE2EDuration="1.876593628s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:26 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:27.875241478 +0000 UTC m=+760.056826024" watchObservedRunningTime="2026-09-18 22:29:27.876593628 +0000 UTC m=+760.058178180"
Sep 18 22:29:28.836112 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:28.836070    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:29.784425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:29.784391    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:29:32.836005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:32.835965    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:38.838545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:38.838499    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:39.568504 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.568454    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:29:39.573941 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.573917    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.576962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.576942    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-44d735\""
Sep 18 22:29:39.584391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.584364    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:29:39.608988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.608956    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:29:39.612918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.612901    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.616378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.616353    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-44d735-dockercfg-6k9c7\""
Sep 18 22:29:39.623664 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.623645    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:29:39.709758 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.709723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nrqvh\" (UniqueName: \"kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.709914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.709772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.709914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.709794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jc97h\" (UniqueName: \"kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.709914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.709863    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.811356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.811321    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nrqvh\" (UniqueName: \"kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.811528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.811375    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.811528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.811404    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jc97h\" (UniqueName: \"kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.811638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.811541    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.812185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.812161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.812322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.812297    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.820629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.820569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jc97h\" (UniqueName: \"kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h\") pod \"payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") " pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:39.820916 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.820895    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nrqvh\" (UniqueName: \"kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh\") pod \"payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.845788 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.845754    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:39.885376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.885347    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:39.924105 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:39.924068    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:40.189506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.189478    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:29:40.192074 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:40.192018    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbaafe798_42d1_4556_9636_25647bafd06b.slice/crio-6a5a26bd81ad8764224dc10530cb93db10e890156356cd0f33b2e1c048cf2daa WatchSource:0}: Error finding container 6a5a26bd81ad8764224dc10530cb93db10e890156356cd0f33b2e1c048cf2daa: Status 404 returned error can't find the container with id 6a5a26bd81ad8764224dc10530cb93db10e890156356cd0f33b2e1c048cf2daa
Sep 18 22:29:40.239924 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.239411    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:29:40.905315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.905277    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" event={"ID":"baafe798-42d1-4556-9636-25647bafd06b","Type":"ContainerStarted","Data":"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"}
Sep 18 22:29:40.905804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.905334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" event={"ID":"baafe798-42d1-4556-9636-25647bafd06b","Type":"ContainerStarted","Data":"6a5a26bd81ad8764224dc10530cb93db10e890156356cd0f33b2e1c048cf2daa"}
Sep 18 22:29:40.905804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.905413    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:40.907078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.907023    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" event={"ID":"6dc073aa-87cf-4a31-ace1-25156fabd404","Type":"ContainerStarted","Data":"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"}
Sep 18 22:29:40.907205 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.907081    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" event={"ID":"6dc073aa-87cf-4a31-ace1-25156fabd404","Type":"ContainerStarted","Data":"c7dedbb4b5ff35b12f904e9a9b976d3f76b28bc289ae08b1dc0336a606d452bc"}
Sep 18 22:29:40.907205 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.907185    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:40.924124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.924067    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" podStartSLOduration=1.924045687 podStartE2EDuration="1.924045687s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:40.923650057 +0000 UTC m=+773.105234598" watchObservedRunningTime="2026-09-18 22:29:40.924045687 +0000 UTC m=+773.105630234"
Sep 18 22:29:40.945185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:40.945113    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" podStartSLOduration=1.9450906730000002 podStartE2EDuration="1.945090673s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:39 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:40.940992986 +0000 UTC m=+773.122577533" watchObservedRunningTime="2026-09-18 22:29:40.945090673 +0000 UTC m=+773.126675221"
Sep 18 22:29:46.204166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:46.204125    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:29:47.947498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.947465    2551 generic.go:356] "Generic (PLEG): container finished" podID="113471d7-5659-42df-9fcc-236e4b38cc20" containerID="529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4" exitCode=0
Sep 18 22:29:47.947946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.947540    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerDied","Data":"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"}
Sep 18 22:29:47.948050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.948014    2551 scope.go:117] "RemoveContainer" containerID="529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"
Sep 18 22:29:47.950436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.950402    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:47.950527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.950489    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:47.950567 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:47.950538    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:48.960018 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:48.959970    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerStarted","Data":"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"}
Sep 18 22:29:49.788118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:49.788081    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:49.789117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:49.789094    2551 patch_prober.go:28] interesting pod/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 22:29:49.789195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:49.789142    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 22:29:50.788774 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:50.788738    2551 patch_prober.go:28] interesting pod/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 22:29:50.789166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:50.788803    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 22:29:51.788379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:51.788343    2551 patch_prober.go:28] interesting pod/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 22:29:51.788568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:51.788418    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 22:29:51.914833 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:51.914796    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:29:51.915215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:51.914993    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:29:52.788182 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:52.788145    2551 patch_prober.go:28] interesting pod/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 22:29:52.788356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:52.788199    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 22:29:52.800294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:52.800270    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:53.788578 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:53.788537    2551 patch_prober.go:28] interesting pod/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused" start-of-body=
Sep 18 22:29:53.788963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:53.788596    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.33:15021/healthz/ready\": dial tcp 10.132.0.33:15021: connect: connection refused"
Sep 18 22:29:54.792293 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:54.792264    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:54.793233 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:54.793213    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:29:55.460405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.460372    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:55.460617 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.460590    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" podUID="5c0003b2-4781-4799-836c-3e17fa243907" containerName="maas-api" containerID="cri-o://28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8" gracePeriod=30
Sep 18 22:29:55.516171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.516136    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:55.516382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.516357    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" podUID="25097b86-a8d4-4a5f-af53-489a937bf956" containerName="payload-processing" containerID="cri-o://4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a" gracePeriod=30
Sep 18 22:29:55.527288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.527256    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:55.527524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.527500    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" podUID="250cfe09-d331-449e-90a2-1ffe01e48022" containerName="payload-pre-processing" containerID="cri-o://68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db" gracePeriod=30
Sep 18 22:29:55.790947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.790919    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:55.831758 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.831723    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:55.842775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.842739    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:55.879095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879057    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-drbwf\" (UniqueName: \"kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf\") pod \"5c0003b2-4781-4799-836c-3e17fa243907\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") "
Sep 18 22:29:55.879272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879166    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sbvxd\" (UniqueName: \"kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd\") pod \"25097b86-a8d4-4a5f-af53-489a937bf956\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") "
Sep 18 22:29:55.879272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879208    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs\") pod \"5c0003b2-4781-4799-836c-3e17fa243907\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") "
Sep 18 22:29:55.879272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879248    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca\") pod \"5c0003b2-4781-4799-836c-3e17fa243907\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") "
Sep 18 22:29:55.879433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879289    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume\") pod \"250cfe09-d331-449e-90a2-1ffe01e48022\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") "
Sep 18 22:29:55.879433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879317    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume\") pod \"25097b86-a8d4-4a5f-af53-489a937bf956\" (UID: \"25097b86-a8d4-4a5f-af53-489a937bf956\") "
Sep 18 22:29:55.879433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879375    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cf77f\" (UniqueName: \"kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f\") pod \"250cfe09-d331-449e-90a2-1ffe01e48022\" (UID: \"250cfe09-d331-449e-90a2-1ffe01e48022\") "
Sep 18 22:29:55.879433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.879413    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls\") pod \"5c0003b2-4781-4799-836c-3e17fa243907\" (UID: \"5c0003b2-4781-4799-836c-3e17fa243907\") "
Sep 18 22:29:55.885188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.881941    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "250cfe09-d331-449e-90a2-1ffe01e48022" (UID: "250cfe09-d331-449e-90a2-1ffe01e48022"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:29:55.885188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.882279    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "25097b86-a8d4-4a5f-af53-489a937bf956" (UID: "25097b86-a8d4-4a5f-af53-489a937bf956"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:29:55.885188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.882702    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "5c0003b2-4781-4799-836c-3e17fa243907" (UID: "5c0003b2-4781-4799-836c-3e17fa243907"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:29:55.886544 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.886459    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:29:55.889165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.889122    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf" (OuterVolumeSpecName: "kube-api-access-drbwf") pod "5c0003b2-4781-4799-836c-3e17fa243907" (UID: "5c0003b2-4781-4799-836c-3e17fa243907"). InnerVolumeSpecName "kube-api-access-drbwf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:55.889487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.889457    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "5c0003b2-4781-4799-836c-3e17fa243907" (UID: "5c0003b2-4781-4799-836c-3e17fa243907"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:29:55.890042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.889986    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f" (OuterVolumeSpecName: "kube-api-access-cf77f") pod "250cfe09-d331-449e-90a2-1ffe01e48022" (UID: "250cfe09-d331-449e-90a2-1ffe01e48022"). InnerVolumeSpecName "kube-api-access-cf77f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:55.891858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.891825    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd" (OuterVolumeSpecName: "kube-api-access-sbvxd") pod "25097b86-a8d4-4a5f-af53-489a937bf956" (UID: "25097b86-a8d4-4a5f-af53-489a937bf956"). InnerVolumeSpecName "kube-api-access-sbvxd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:29:55.893646 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.893623    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "5c0003b2-4781-4799-836c-3e17fa243907" (UID: "5c0003b2-4781-4799-836c-3e17fa243907"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:29:55.980094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980058    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/250cfe09-d331-449e-90a2-1ffe01e48022-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980088    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/25097b86-a8d4-4a5f-af53-489a937bf956-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980098    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cf77f\" (UniqueName: \"kubernetes.io/projected/250cfe09-d331-449e-90a2-1ffe01e48022-kube-api-access-cf77f\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980108    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-maas-api-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980117    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-drbwf\" (UniqueName: \"kubernetes.io/projected/5c0003b2-4781-4799-836c-3e17fa243907-kube-api-access-drbwf\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980127    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sbvxd\" (UniqueName: \"kubernetes.io/projected/25097b86-a8d4-4a5f-af53-489a937bf956-kube-api-access-sbvxd\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980135    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/5c0003b2-4781-4799-836c-3e17fa243907-metrics-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.980373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.980144    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/5c0003b2-4781-4799-836c-3e17fa243907-openshift-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:29:55.998240 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.998204    2551 generic.go:356] "Generic (PLEG): container finished" podID="5c0003b2-4781-4799-836c-3e17fa243907" containerID="28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8" exitCode=0
Sep 18 22:29:55.998407 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.998258    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"
Sep 18 22:29:55.998407 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.998295    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" event={"ID":"5c0003b2-4781-4799-836c-3e17fa243907","Type":"ContainerDied","Data":"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"}
Sep 18 22:29:55.998407 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.998352    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv" event={"ID":"5c0003b2-4781-4799-836c-3e17fa243907","Type":"ContainerDied","Data":"a746efdd2cede39e0a6139ebf8ded0bca379195627df32a749ee01b159d81e48"}
Sep 18 22:29:55.998407 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.998384    2551 scope.go:117] "RemoveContainer" containerID="28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"
Sep 18 22:29:55.999686 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.999665    2551 generic.go:356] "Generic (PLEG): container finished" podID="250cfe09-d331-449e-90a2-1ffe01e48022" containerID="68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db" exitCode=0
Sep 18 22:29:55.999760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.999710    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"
Sep 18 22:29:55.999760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.999715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" event={"ID":"250cfe09-d331-449e-90a2-1ffe01e48022","Type":"ContainerDied","Data":"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"}
Sep 18 22:29:55.999851 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:55.999792    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t" event={"ID":"250cfe09-d331-449e-90a2-1ffe01e48022","Type":"ContainerDied","Data":"abaf3078e910acbfe1af7bc07da9b1bea13de056c6ca89384d5a1a594a6cb092"}
Sep 18 22:29:56.001509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.001392    2551 generic.go:356] "Generic (PLEG): container finished" podID="25097b86-a8d4-4a5f-af53-489a937bf956" containerID="4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a" exitCode=0
Sep 18 22:29:56.001509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.001429    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"
Sep 18 22:29:56.001509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.001482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" event={"ID":"25097b86-a8d4-4a5f-af53-489a937bf956","Type":"ContainerDied","Data":"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"}
Sep 18 22:29:56.001748 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.001523    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs" event={"ID":"25097b86-a8d4-4a5f-af53-489a937bf956","Type":"ContainerDied","Data":"3f8f168293cd140a53951ffcb45535babc5a395d96599b78c25380e21b847390"}
Sep 18 22:29:56.001748 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.001725    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy" containerID="cri-o://845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b" gracePeriod=30
Sep 18 22:29:56.009394 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.009377    2551 scope.go:117] "RemoveContainer" containerID="28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"
Sep 18 22:29:56.009731 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:56.009706    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8\": container with ID starting with 28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8 not found: ID does not exist" containerID="28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"
Sep 18 22:29:56.009838 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.009740    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8"} err="failed to get container status \"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8\": rpc error: code = NotFound desc = could not find container \"28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8\": container with ID starting with 28a4c77d2da099ed751e33123da3c75ca1c9eee30034ad36923b3f1e0a4959c8 not found: ID does not exist"
Sep 18 22:29:56.009838 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.009763    2551 scope.go:117] "RemoveContainer" containerID="68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"
Sep 18 22:29:56.018003 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.017982    2551 scope.go:117] "RemoveContainer" containerID="68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"
Sep 18 22:29:56.018270 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:56.018252    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db\": container with ID starting with 68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db not found: ID does not exist" containerID="68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"
Sep 18 22:29:56.018327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.018276    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db"} err="failed to get container status \"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db\": rpc error: code = NotFound desc = could not find container \"68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db\": container with ID starting with 68bf691b2c985831f82cd421484c547cec08f82cf24ec0ffd133d9a5e3bc33db not found: ID does not exist"
Sep 18 22:29:56.018327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.018292    2551 scope.go:117] "RemoveContainer" containerID="4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"
Sep 18 22:29:56.027716 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.026246    2551 scope.go:117] "RemoveContainer" containerID="4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"
Sep 18 22:29:56.027716 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:29:56.027406    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a\": container with ID starting with 4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a not found: ID does not exist" containerID="4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"
Sep 18 22:29:56.027716 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.027435    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a"} err="failed to get container status \"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a\": rpc error: code = NotFound desc = could not find container \"4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a\": container with ID starting with 4150930eb11908404077117dc7b2e8761e6a1a9da47d37b667e152071cdc7b2a not found: ID does not exist"
Sep 18 22:29:56.044460 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.044432    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:56.048114 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.048087    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-519912e2-557989fc8b-8ph5t"]
Sep 18 22:29:56.063431 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.063397    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:56.064510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.064474    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-519912e2-66fc55f5d5-6k8lv"]
Sep 18 22:29:56.074362 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.074333    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:56.080489 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.080457    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-519912e2-66ff84bcf-kqbvs"]
Sep 18 22:29:56.291544 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.291453    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="25097b86-a8d4-4a5f-af53-489a937bf956" path="/var/lib/kubelet/pods/25097b86-a8d4-4a5f-af53-489a937bf956/volumes"
Sep 18 22:29:56.291917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.291897    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="250cfe09-d331-449e-90a2-1ffe01e48022" path="/var/lib/kubelet/pods/250cfe09-d331-449e-90a2-1ffe01e48022/volumes"
Sep 18 22:29:56.292349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.292329    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5c0003b2-4781-4799-836c-3e17fa243907" path="/var/lib/kubelet/pods/5c0003b2-4781-4799-836c-3e17fa243907/volumes"
Sep 18 22:29:56.832893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.832854    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:29:56.833445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833427    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="250cfe09-d331-449e-90a2-1ffe01e48022" containerName="payload-pre-processing"
Sep 18 22:29:56.833489 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833449    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="250cfe09-d331-449e-90a2-1ffe01e48022" containerName="payload-pre-processing"
Sep 18 22:29:56.833489 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833484    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="25097b86-a8d4-4a5f-af53-489a937bf956" containerName="payload-processing"
Sep 18 22:29:56.833549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833493    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="25097b86-a8d4-4a5f-af53-489a937bf956" containerName="payload-processing"
Sep 18 22:29:56.833549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833513    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5c0003b2-4781-4799-836c-3e17fa243907" containerName="maas-api"
Sep 18 22:29:56.833549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833522    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5c0003b2-4781-4799-836c-3e17fa243907" containerName="maas-api"
Sep 18 22:29:56.833641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833626    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="250cfe09-d331-449e-90a2-1ffe01e48022" containerName="payload-pre-processing"
Sep 18 22:29:56.833673 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833640    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="25097b86-a8d4-4a5f-af53-489a937bf956" containerName="payload-processing"
Sep 18 22:29:56.833673 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.833651    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5c0003b2-4781-4799-836c-3e17fa243907" containerName="maas-api"
Sep 18 22:29:56.838258 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.838236    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.849879 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.849852    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"kube-root-ca.crt\""
Sep 18 22:29:56.850739 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.850723    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"simulator-w0-c5fdd3-kserve-self-signed-certs\""
Sep 18 22:29:56.857855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.857832    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"openshift-service-ca.crt\""
Sep 18 22:29:56.858078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.858062    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"default-dockercfg-x8nc2\""
Sep 18 22:29:56.870483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.870453    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:29:56.872432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.872409    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:29:56.874877 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.874862    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.881859 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.881840    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-migrate-67d98638-gw-openshift-default-dockercfg-wwsq6\""
Sep 18 22:29:56.887674 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887651    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.887763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887694    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.887763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887716    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.887763 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887748    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.887890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887773    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.887890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887792    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qthrb\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.887890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887842    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.887890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887874    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.888048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887924    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.888048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887942    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n229q\" (UniqueName: \"kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.888048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887980    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.888048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.887995    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.888048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.888013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.888286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.888079    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.888286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.888178    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.899599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.899576    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:29:56.988634 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988634 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988658    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988681    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988707    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qthrb\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988841    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n229q\" (UniqueName: \"kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.988900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988879    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988905    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988937    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.988970    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989001    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989119    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989560    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989635    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.989808 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989742    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989808 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.989878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.989838    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.990197 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.990170    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.991627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.991602    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.991749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.991699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:56.991811 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.991795    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:56.991863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:56.991805    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:57.000572 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.000548    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n229q\" (UniqueName: \"kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q\") pod \"simulator-w0-c5fdd3-kserve-7879f97864-5vxs6\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") " pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:57.004905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.004885    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qthrb\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:57.012547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.012522    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token\") pod \"e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") " pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:57.150415 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.150379    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:29:57.187970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.187932    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:57.431473 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.431446    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:29:57.433453 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:57.433421    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5b57a4a7_e7a6_4d87_90d5_cd011aa46c43.slice/crio-55cdc1b54e27e6c164cdb0321e92e253dfd7e0ad8001f679f4aebf361c2fef50 WatchSource:0}: Error finding container 55cdc1b54e27e6c164cdb0321e92e253dfd7e0ad8001f679f4aebf361c2fef50: Status 404 returned error can't find the container with id 55cdc1b54e27e6c164cdb0321e92e253dfd7e0ad8001f679f4aebf361c2fef50
Sep 18 22:29:57.477421 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.477391    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:29:57.479396 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:29:57.479369    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc4988f2f_9e13_4319_9632_9618ce5371e3.slice/crio-b0d7ddd61f0009d671fded8158a3f6135285f971e8141d5e2f8c4ae764e7e5e8 WatchSource:0}: Error finding container b0d7ddd61f0009d671fded8158a3f6135285f971e8141d5e2f8c4ae764e7e5e8: Status 404 returned error can't find the container with id b0d7ddd61f0009d671fded8158a3f6135285f971e8141d5e2f8c4ae764e7e5e8
Sep 18 22:29:57.482402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.482325    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:57.482490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.482414    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:57.482490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:57.482448    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:29:58.018944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.018905    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerStarted","Data":"8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d"}
Sep 18 22:29:58.019395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.018971    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerStarted","Data":"b0d7ddd61f0009d671fded8158a3f6135285f971e8141d5e2f8c4ae764e7e5e8"}
Sep 18 22:29:58.020545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.020521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" event={"ID":"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43","Type":"ContainerStarted","Data":"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"}
Sep 18 22:29:58.020627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.020553    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" event={"ID":"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43","Type":"ContainerStarted","Data":"55cdc1b54e27e6c164cdb0321e92e253dfd7e0ad8001f679f4aebf361c2fef50"}
Sep 18 22:29:58.042980 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.042926    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podStartSLOduration=2.04290841 podStartE2EDuration="2.04290841s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:58.041955672 +0000 UTC m=+790.223540218" watchObservedRunningTime="2026-09-18 22:29:58.04290841 +0000 UTC m=+790.224492955"
Sep 18 22:29:58.067797 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.067743    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" podStartSLOduration=2.067728652 podStartE2EDuration="2.067728652s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:29:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:29:58.06577627 +0000 UTC m=+790.247360816" watchObservedRunningTime="2026-09-18 22:29:58.067728652 +0000 UTC m=+790.249313197"
Sep 18 22:29:58.188518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.188485    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:58.194520 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:58.194495    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:59.026088 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:59.026059    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:29:59.027097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:29:59.027079    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:00.150472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.150437    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"]
Sep 18 22:30:00.152962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.152944    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"]
Sep 18 22:30:00.153103 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.153084    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"
Sep 18 22:30:00.155669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.155643    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"]
Sep 18 22:30:00.155804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.155780    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"
Sep 18 22:30:00.156893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.156871    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:30:00.157985 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.157968    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"
Sep 18 22:30:00.164376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.164359    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"
Sep 18 22:30:00.169186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.168978    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"]
Sep 18 22:30:00.174504 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.174478    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"
Sep 18 22:30:00.178243 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.178008    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"]
Sep 18 22:30:00.180598 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.180568    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"]
Sep 18 22:30:00.221513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.221482    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"
Sep 18 22:30:00.460836 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.460758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"]
Sep 18 22:30:00.463953 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:00.463921    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92afe595_570c_4df2_ac32_562c2e0bc798.slice/crio-5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664 WatchSource:0}: Error finding container 5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664: Status 404 returned error can't find the container with id 5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664
Sep 18 22:30:00.490642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.490619    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"]
Sep 18 22:30:00.491951 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:00.491928    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod37ed21b9_d433_4923_a913_db5ed5844367.slice/crio-3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956 WatchSource:0}: Error finding container 3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956: Status 404 returned error can't find the container with id 3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956
Sep 18 22:30:00.539317 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:00.539210    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"]
Sep 18 22:30:00.542063 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:00.542015    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89b4f896_ecf4_477e_ab9f_2cd8e66dced5.slice/crio-a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463 WatchSource:0}: Error finding container a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463: Status 404 returned error can't find the container with id a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463
Sep 18 22:30:01.040020 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.039977    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr" event={"ID":"37ed21b9-d433-4923-a913-db5ed5844367","Type":"ContainerStarted","Data":"3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956"}
Sep 18 22:30:01.041354 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.041328    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4" event={"ID":"92afe595-570c-4df2-ac32-562c2e0bc798","Type":"ContainerStarted","Data":"5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664"}
Sep 18 22:30:01.042480 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.042456    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq" event={"ID":"89b4f896-ecf4-477e-ab9f-2cd8e66dced5","Type":"ContainerStarted","Data":"a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463"}
Sep 18 22:30:01.260670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.260632    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:30:01.332386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332295    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wgcfg\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332387    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332428    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332466    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332514    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332549    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332803 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332600    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332803 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332633    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.332803 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.332659    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy\") pod \"113471d7-5659-42df-9fcc-236e4b38cc20\" (UID: \"113471d7-5659-42df-9fcc-236e4b38cc20\") "
Sep 18 22:30:01.333720 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.333331    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:01.334214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.334161    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:01.334396 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.334376    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:01.334465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.334431    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data" (OuterVolumeSpecName: "istio-data") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:01.334639 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.334604    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:30:01.339736 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.339642    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:30:01.344542 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.344457    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg" (OuterVolumeSpecName: "kube-api-access-wgcfg") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "kube-api-access-wgcfg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:01.346586 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.346396    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token" (OuterVolumeSpecName: "istio-token") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:01.346975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.346953    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "113471d7-5659-42df-9fcc-236e4b38cc20" (UID: "113471d7-5659-42df-9fcc-236e4b38cc20"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434424    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434462    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wgcfg\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-kube-api-access-wgcfg\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434477    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434490    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/113471d7-5659-42df-9fcc-236e4b38cc20-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434503    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434516    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434529    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/113471d7-5659-42df-9fcc-236e4b38cc20-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434542    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/113471d7-5659-42df-9fcc-236e4b38cc20-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:01.434580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:01.434555    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/113471d7-5659-42df-9fcc-236e4b38cc20-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:02.054148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.053322    2551 generic.go:356] "Generic (PLEG): container finished" podID="113471d7-5659-42df-9fcc-236e4b38cc20" containerID="845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b" exitCode=0
Sep 18 22:30:02.054148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.053375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerDied","Data":"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"}
Sep 18 22:30:02.054148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.053429    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2" event={"ID":"113471d7-5659-42df-9fcc-236e4b38cc20","Type":"ContainerDied","Data":"d1cf68de137483033650ca6138f293c481c5cace19f84f6ac960954652594458"}
Sep 18 22:30:02.054148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.053463    2551 scope.go:117] "RemoveContainer" containerID="845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"
Sep 18 22:30:02.054148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.053626    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"
Sep 18 22:30:02.091811 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.091773    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:30:02.105351 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.105316    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-519912e2-openshift-default-866d855bc6-27sg2"]
Sep 18 22:30:02.292518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.292476    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" path="/var/lib/kubelet/pods/113471d7-5659-42df-9fcc-236e4b38cc20/volumes"
Sep 18 22:30:02.480654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.480619    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:02.481135 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.481113    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:02.481135 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.481133    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:02.481263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.481213    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:02.486614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.486589    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.490152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.490128    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-migrate-67d98638-serving-cert\""
Sep 18 22:30:02.501841 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.501815    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:02.522762 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.522703    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:30:02.546315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.546278    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.546484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.546338    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.546484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.546470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.546627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.546537    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r6mnh\" (UniqueName: \"kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.648100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.648055    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.648283 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.648156    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.648283 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:02.648213    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-serving-cert: secret "maas-api-e2e-migrate-67d98638-serving-cert" not found
Sep 18 22:30:02.648283 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:02.648280    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls podName:a8885dba-2e31-4b13-a57f-9fc5635aff6e nodeName:}" failed. No retries permitted until 2026-09-18 22:30:03.148258548 +0000 UTC m=+795.329843093 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls") pod "maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e") : secret "maas-api-e2e-migrate-67d98638-serving-cert" not found
Sep 18 22:30:02.648462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.648212    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r6mnh\" (UniqueName: \"kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.648462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.648406    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.649202 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.649174    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.651140 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.651118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.659844 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.659823    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-r6mnh\" (UniqueName: \"kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:02.777272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.777000    2551 scope.go:117] "RemoveContainer" containerID="529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"
Sep 18 22:30:02.785636 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.785617    2551 scope.go:117] "RemoveContainer" containerID="845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"
Sep 18 22:30:02.785932 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:02.785913    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b\": container with ID starting with 845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b not found: ID does not exist" containerID="845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"
Sep 18 22:30:02.785981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.785939    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b"} err="failed to get container status \"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b\": rpc error: code = NotFound desc = could not find container \"845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b\": container with ID starting with 845e793a669d02eb56668b46e0bf8ba471e494f08f103268bf056a9a2a989e3b not found: ID does not exist"
Sep 18 22:30:02.785981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.785956    2551 scope.go:117] "RemoveContainer" containerID="529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"
Sep 18 22:30:02.786237 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:02.786220    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4\": container with ID starting with 529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4 not found: ID does not exist" containerID="529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"
Sep 18 22:30:02.786300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:02.786242    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4"} err="failed to get container status \"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4\": rpc error: code = NotFound desc = could not find container \"529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4\": container with ID starting with 529704a468248402e45234467bbcd136885d6d1a0c01be8869086be896118fa4 not found: ID does not exist"
Sep 18 22:30:03.155075 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.155013    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:03.155178 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:03.155146    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-serving-cert: secret "maas-api-e2e-migrate-67d98638-serving-cert" not found
Sep 18 22:30:03.155215 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:03.155205    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls podName:a8885dba-2e31-4b13-a57f-9fc5635aff6e nodeName:}" failed. No retries permitted until 2026-09-18 22:30:04.155190337 +0000 UTC m=+796.336774865 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls") pod "maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e") : secret "maas-api-e2e-migrate-67d98638-serving-cert" not found
Sep 18 22:30:03.409225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.409192    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:03.409667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.409652    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:03.409667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.409667    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:03.409757 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.409728    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="113471d7-5659-42df-9fcc-236e4b38cc20" containerName="istio-proxy"
Sep 18 22:30:03.412972 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.412952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.416154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.416127    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-67d98638\""
Sep 18 22:30:03.425604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.425579    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:03.458006 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.457975    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.458250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.458010    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m9fwd\" (UniqueName: \"kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.558757 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.558717    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.558757 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.558756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m9fwd\" (UniqueName: \"kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.559406 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.559384    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.567998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.567974    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-m9fwd\" (UniqueName: \"kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd\") pod \"payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.723020 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.722926    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:03.999315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:03.999246    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:04.003963 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:04.003941    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6dd69384_92ae_49fb_b1ae_906e917554b3.slice/crio-4c9d655307ef65a717ac064e7d0d1cb0ad7ada6cae7a5c39893a05c1f55ba328 WatchSource:0}: Error finding container 4c9d655307ef65a717ac064e7d0d1cb0ad7ada6cae7a5c39893a05c1f55ba328: Status 404 returned error can't find the container with id 4c9d655307ef65a717ac064e7d0d1cb0ad7ada6cae7a5c39893a05c1f55ba328
Sep 18 22:30:04.067950 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.067919    2551 generic.go:356] "Generic (PLEG): container finished" podID="92afe595-570c-4df2-ac32-562c2e0bc798" containerID="49bfc854e51875cc537742f85938532991d6029f19275d0b55ddf1e6cf6969ac" exitCode=0
Sep 18 22:30:04.068118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.067987    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4" event={"ID":"92afe595-570c-4df2-ac32-562c2e0bc798","Type":"ContainerDied","Data":"49bfc854e51875cc537742f85938532991d6029f19275d0b55ddf1e6cf6969ac"}
Sep 18 22:30:04.069658 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.069634    2551 generic.go:356] "Generic (PLEG): container finished" podID="89b4f896-ecf4-477e-ab9f-2cd8e66dced5" containerID="df3ab15bb5d7cc1bf7c14b1ed09e964a939cbca5d9c441740836fd8a46ee598c" exitCode=0
Sep 18 22:30:04.069789 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.069715    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq" event={"ID":"89b4f896-ecf4-477e-ab9f-2cd8e66dced5","Type":"ContainerDied","Data":"df3ab15bb5d7cc1bf7c14b1ed09e964a939cbca5d9c441740836fd8a46ee598c"}
Sep 18 22:30:04.071339 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.071318    2551 generic.go:356] "Generic (PLEG): container finished" podID="37ed21b9-d433-4923-a913-db5ed5844367" containerID="7278ee0d515de9aa581e1f910c2f326add93ffbe891657ae6fc5e7945814e98b" exitCode=0
Sep 18 22:30:04.071436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.071344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr" event={"ID":"37ed21b9-d433-4923-a913-db5ed5844367","Type":"ContainerDied","Data":"7278ee0d515de9aa581e1f910c2f326add93ffbe891657ae6fc5e7945814e98b"}
Sep 18 22:30:04.072534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.072513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" event={"ID":"6dd69384-92ae-49fb-b1ae-906e917554b3","Type":"ContainerStarted","Data":"4c9d655307ef65a717ac064e7d0d1cb0ad7ada6cae7a5c39893a05c1f55ba328"}
Sep 18 22:30:04.165073 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.165025    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:04.167537 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.167503    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") " pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:04.302291 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.302206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:04.576876 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:04.576846    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:04.577912 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:04.577884    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8885dba_2e31_4b13_a57f_9fc5635aff6e.slice/crio-45ac6b617767bce969749f202bfd7b34c29dbcfbcb0eadfee152e9a83a0e2861 WatchSource:0}: Error finding container 45ac6b617767bce969749f202bfd7b34c29dbcfbcb0eadfee152e9a83a0e2861: Status 404 returned error can't find the container with id 45ac6b617767bce969749f202bfd7b34c29dbcfbcb0eadfee152e9a83a0e2861
Sep 18 22:30:05.080012 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.079980    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" event={"ID":"6dd69384-92ae-49fb-b1ae-906e917554b3","Type":"ContainerStarted","Data":"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"}
Sep 18 22:30:05.080195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.080179    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:05.081799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.081774    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" event={"ID":"a8885dba-2e31-4b13-a57f-9fc5635aff6e","Type":"ContainerStarted","Data":"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"}
Sep 18 22:30:05.081896 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.081836    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:05.081896 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.081874    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" event={"ID":"a8885dba-2e31-4b13-a57f-9fc5635aff6e","Type":"ContainerStarted","Data":"45ac6b617767bce969749f202bfd7b34c29dbcfbcb0eadfee152e9a83a0e2861"}
Sep 18 22:30:05.100175 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.100077    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" podStartSLOduration=2.100012334 podStartE2EDuration="2.100012334s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:05.097366203 +0000 UTC m=+797.278950814" watchObservedRunningTime="2026-09-18 22:30:05.100012334 +0000 UTC m=+797.281596883"
Sep 18 22:30:05.118014 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.117702    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" podStartSLOduration=2.737733583 podStartE2EDuration="3.117682575s" totalImagesPullingTime="379.948992ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:05.116391763 +0000 UTC m=+797.297976309" watchObservedRunningTime="2026-09-18 22:30:05.117682575 +0000 UTC m=+797.299267119"
Sep 18 22:30:05.230799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.230715    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"
Sep 18 22:30:05.255339 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.255310    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"
Sep 18 22:30:05.270086 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:05.270058    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"
Sep 18 22:30:06.088669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.088584    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr"
Sep 18 22:30:06.088669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.088612    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829510-hkklr" event={"ID":"37ed21b9-d433-4923-a913-db5ed5844367","Type":"ContainerDied","Data":"3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956"}
Sep 18 22:30:06.088669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.088662    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3abd0414f98f83c58a1e39cab1e7efdfcd60c1625cf2bba470c704a603b5e956"
Sep 18 22:30:06.089858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.089839    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"
Sep 18 22:30:06.089975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.089874    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4" event={"ID":"92afe595-570c-4df2-ac32-562c2e0bc798","Type":"ContainerDied","Data":"5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664"}
Sep 18 22:30:06.089975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.089922    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fbc855f7d4fce4fe2bdf8d81d5700cdbcf6fb81a4cc990883b028ed70a17664"
Sep 18 22:30:06.091218 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.091194    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq" event={"ID":"89b4f896-ecf4-477e-ab9f-2cd8e66dced5","Type":"ContainerDied","Data":"a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463"}
Sep 18 22:30:06.091298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.091232    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a2d94699bbc50fedff64cb3aba161132397e1ef455447da663a8345989b35463"
Sep 18 22:30:06.091298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:06.091206    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"
Sep 18 22:30:07.151099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:07.151058    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:30:07.151099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:07.151105    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:30:07.164267 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:07.164235    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:30:08.113899 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:08.113867    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:30:11.097416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:11.097338    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:16.093524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:16.093489    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:21.191632 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:21.191591    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:30:23.195956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.195922    2551 generic.go:356] "Generic (PLEG): container finished" podID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerID="8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d" exitCode=0
Sep 18 22:30:23.196474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.195961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerDied","Data":"8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d"}
Sep 18 22:30:23.196474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.196438    2551 scope.go:117] "RemoveContainer" containerID="8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d"
Sep 18 22:30:23.197613 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.197578    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:23.197692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.197652    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:23.197692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:23.197683    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:24.203285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:24.203250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerStarted","Data":"0e139c60d8b787312a06f19b3bd5895ccbdf58236dd9bdcb4d002ae1b9dae5d9"}
Sep 18 22:30:25.188855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:25.188815    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:25.189950 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:25.189923    2551 patch_prober.go:28] interesting pod/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 18 22:30:25.190084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:25.189972    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 18 22:30:26.188487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:26.188453    2551 patch_prober.go:28] interesting pod/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 18 22:30:26.188867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:26.188511    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 18 22:30:27.188391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:27.188354    2551 patch_prober.go:28] interesting pod/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 18 22:30:27.188565 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:27.188413    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 18 22:30:28.188466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:28.188437    2551 patch_prober.go:28] interesting pod/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused" start-of-body=
Sep 18 22:30:28.188699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:28.188675    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.40:15021/healthz/ready\": dial tcp 10.132.0.40:15021: connect: connection refused"
Sep 18 22:30:29.026337 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.026296    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:29.350215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.350119    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:29.351957 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.351929    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:29.613773 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.613690    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"]
Sep 18 22:30:29.614127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614113    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89b4f896-ecf4-477e-ab9f-2cd8e66dced5" containerName="cleanup"
Sep 18 22:30:29.614180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614132    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="89b4f896-ecf4-477e-ab9f-2cd8e66dced5" containerName="cleanup"
Sep 18 22:30:29.614180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614145    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="37ed21b9-d433-4923-a913-db5ed5844367" containerName="cleanup"
Sep 18 22:30:29.614180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614151    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="37ed21b9-d433-4923-a913-db5ed5844367" containerName="cleanup"
Sep 18 22:30:29.614180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614167    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92afe595-570c-4df2-ac32-562c2e0bc798" containerName="cleanup"
Sep 18 22:30:29.614180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614172    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="92afe595-570c-4df2-ac32-562c2e0bc798" containerName="cleanup"
Sep 18 22:30:29.614344 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614243    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="92afe595-570c-4df2-ac32-562c2e0bc798" containerName="cleanup"
Sep 18 22:30:29.614344 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614252    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="89b4f896-ecf4-477e-ab9f-2cd8e66dced5" containerName="cleanup"
Sep 18 22:30:29.614344 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.614260    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="37ed21b9-d433-4923-a913-db5ed5844367" containerName="cleanup"
Sep 18 22:30:29.620286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.620264    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:29.623179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.623158    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:30:29.624906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.624877    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"]
Sep 18 22:30:29.694825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.694797    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9\" (UID: \"80c21ac7-d6c8-48f2-95ec-e04d5548e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:29.795871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.795842    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9\" (UID: \"80c21ac7-d6c8-48f2-95ec-e04d5548e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:29.796641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.796612    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9\" (UID: \"80c21ac7-d6c8-48f2-95ec-e04d5548e13e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:29.932910 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:29.932882    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:30.206918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.206894    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"]
Sep 18 22:30:30.208890 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:30.208861    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod80c21ac7_d6c8_48f2_95ec_e04d5548e13e.slice/crio-b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3 WatchSource:0}: Error finding container b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3: Status 404 returned error can't find the container with id b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3
Sep 18 22:30:30.239081 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.239056    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9" event={"ID":"80c21ac7-d6c8-48f2-95ec-e04d5548e13e","Type":"ContainerStarted","Data":"b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3"}
Sep 18 22:30:30.452380 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.452353    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:30:30.457055 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.456992    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.460712 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.460685    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f9cf33\"/\"simulator-w4-f9cf33-kserve-self-signed-certs\""
Sep 18 22:30:30.460712 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.460694    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f9cf33\"/\"kube-root-ca.crt\""
Sep 18 22:30:30.460712 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.460688    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w4-f9cf33\"/\"openshift-service-ca.crt\""
Sep 18 22:30:30.460907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.460688    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w4-f9cf33\"/\"default-dockercfg-qlm4m\""
Sep 18 22:30:30.467967 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.467945    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:30:30.603817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.603784    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.603938 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.603840    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.603938 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.603889    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.603938 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.603906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qtdl9\" (UniqueName: \"kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.604054 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.604024    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.604097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.604061    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705271    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qtdl9\" (UniqueName: \"kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705338    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705353    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705381    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705414    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705435    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705872 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705845    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705884    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.705994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.705894    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.707715 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.707666    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.707847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.707832    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.713513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.713494    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qtdl9\" (UniqueName: \"kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9\") pod \"simulator-w4-f9cf33-kserve-69b7dccf-v6hrl\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") " pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:30.765398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:30.765370    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:31.046284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.046261    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:30:31.092873 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:31.092840    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod64349f74_2f8c_4f35_962a_2c27f88dd31a.slice/crio-2c3f7db33540db4a5e8e7db6824aa56d3f3f3135ed173bac06ad6360b62e13fb WatchSource:0}: Error finding container 2c3f7db33540db4a5e8e7db6824aa56d3f3f3135ed173bac06ad6360b62e13fb: Status 404 returned error can't find the container with id 2c3f7db33540db4a5e8e7db6824aa56d3f3f3135ed173bac06ad6360b62e13fb
Sep 18 22:30:31.246956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.246919    2551 generic.go:356] "Generic (PLEG): container finished" podID="80c21ac7-d6c8-48f2-95ec-e04d5548e13e" containerID="ed98a95554f207b32b894c064cdb66c43e3e24b6bd09ac6920aa8f1df7fca138" exitCode=0
Sep 18 22:30:31.247126 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.247004    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9" event={"ID":"80c21ac7-d6c8-48f2-95ec-e04d5548e13e","Type":"ContainerDied","Data":"ed98a95554f207b32b894c064cdb66c43e3e24b6bd09ac6920aa8f1df7fca138"}
Sep 18 22:30:31.248751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.248729    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" event={"ID":"64349f74-2f8c-4f35-962a-2c27f88dd31a","Type":"ContainerStarted","Data":"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"}
Sep 18 22:30:31.248849 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.248760    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" event={"ID":"64349f74-2f8c-4f35-962a-2c27f88dd31a","Type":"ContainerStarted","Data":"2c3f7db33540db4a5e8e7db6824aa56d3f3f3135ed173bac06ad6360b62e13fb"}
Sep 18 22:30:31.278251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:31.278205    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" podStartSLOduration=1.278191471 podStartE2EDuration="1.278191471s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:31.277402818 +0000 UTC m=+823.458987363" watchObservedRunningTime="2026-09-18 22:30:31.278191471 +0000 UTC m=+823.459776015"
Sep 18 22:30:32.384340 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:32.384317    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:32.522214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:32.522137    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca\") pod \"80c21ac7-d6c8-48f2-95ec-e04d5548e13e\" (UID: \"80c21ac7-d6c8-48f2-95ec-e04d5548e13e\") "
Sep 18 22:30:32.522463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:32.522441    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "80c21ac7-d6c8-48f2-95ec-e04d5548e13e" (UID: "80c21ac7-d6c8-48f2-95ec-e04d5548e13e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:30:32.623529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:32.623504    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/80c21ac7-d6c8-48f2-95ec-e04d5548e13e-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:33.260320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:33.260285    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9" event={"ID":"80c21ac7-d6c8-48f2-95ec-e04d5548e13e","Type":"ContainerDied","Data":"b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3"}
Sep 18 22:30:33.260320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:33.260317    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"
Sep 18 22:30:33.260547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:33.260334    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b41218d42971a853fbd0b2195eab10c92fe4c16582baa77c94c62ef27475fbe3"
Sep 18 22:30:38.990771 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:38.990733    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:30:40.733817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.733782    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:40.734301 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.734047    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" podUID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" containerName="maas-api" containerID="cri-o://a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9" gracePeriod=30
Sep 18 22:30:40.766196 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.766168    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:40.766294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.766214    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:40.781140 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.781112    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:40.784515 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.784488    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:40.784705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.784684    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" podUID="6dd69384-92ae-49fb-b1ae-906e917554b3" containerName="payload-pre-processing" containerID="cri-o://6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf" gracePeriod=30
Sep 18 22:30:40.944382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.943903    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"]
Sep 18 22:30:40.946908 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:40.946873    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-67d98638-qr4l9"]
Sep 18 22:30:41.052254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.052230    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:41.056390 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.055869    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101426    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-m9fwd\" (UniqueName: \"kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd\") pod \"6dd69384-92ae-49fb-b1ae-906e917554b3\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") "
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101479    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") pod \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") "
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101524    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume\") pod \"6dd69384-92ae-49fb-b1ae-906e917554b3\" (UID: \"6dd69384-92ae-49fb-b1ae-906e917554b3\") "
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101606    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca\") pod \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") "
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101630    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-r6mnh\" (UniqueName: \"kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh\") pod \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") "
Sep 18 22:30:41.104050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.101653    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs\") pod \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\" (UID: \"a8885dba-2e31-4b13-a57f-9fc5635aff6e\") "
Sep 18 22:30:41.110092 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.110063    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:30:41.110486 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.110455    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" containerID="cri-o://0e139c60d8b787312a06f19b3bd5895ccbdf58236dd9bdcb4d002ae1b9dae5d9" gracePeriod=30
Sep 18 22:30:41.111848 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.111821    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dd69384-92ae-49fb-b1ae-906e917554b3" (UID: "6dd69384-92ae-49fb-b1ae-906e917554b3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:30:41.113499 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.112880    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "a8885dba-2e31-4b13-a57f-9fc5635aff6e" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:30:41.113609 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.113550    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "a8885dba-2e31-4b13-a57f-9fc5635aff6e" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:30:41.114296 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.114266    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "a8885dba-2e31-4b13-a57f-9fc5635aff6e" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:30:41.116311 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.116267    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh" (OuterVolumeSpecName: "kube-api-access-r6mnh") pod "a8885dba-2e31-4b13-a57f-9fc5635aff6e" (UID: "a8885dba-2e31-4b13-a57f-9fc5635aff6e"). InnerVolumeSpecName "kube-api-access-r6mnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:41.116311 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.116297    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd" (OuterVolumeSpecName: "kube-api-access-m9fwd") pod "6dd69384-92ae-49fb-b1ae-906e917554b3" (UID: "6dd69384-92ae-49fb-b1ae-906e917554b3"). InnerVolumeSpecName "kube-api-access-m9fwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:41.202640 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202614    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m9fwd\" (UniqueName: \"kubernetes.io/projected/6dd69384-92ae-49fb-b1ae-906e917554b3-kube-api-access-m9fwd\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.202726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202643    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-maas-api-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.202726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202654    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dd69384-92ae-49fb-b1ae-906e917554b3-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.202726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202662    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a8885dba-2e31-4b13-a57f-9fc5635aff6e-openshift-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.202726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202671    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r6mnh\" (UniqueName: \"kubernetes.io/projected/a8885dba-2e31-4b13-a57f-9fc5635aff6e-kube-api-access-r6mnh\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.202726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.202679    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a8885dba-2e31-4b13-a57f-9fc5635aff6e-metrics-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:41.304123 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.304048    2551 generic.go:356] "Generic (PLEG): container finished" podID="6dd69384-92ae-49fb-b1ae-906e917554b3" containerID="6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf" exitCode=0
Sep 18 22:30:41.304123 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.304096    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"
Sep 18 22:30:41.304305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.304128    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" event={"ID":"6dd69384-92ae-49fb-b1ae-906e917554b3","Type":"ContainerDied","Data":"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"}
Sep 18 22:30:41.304305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.304177    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg" event={"ID":"6dd69384-92ae-49fb-b1ae-906e917554b3","Type":"ContainerDied","Data":"4c9d655307ef65a717ac064e7d0d1cb0ad7ada6cae7a5c39893a05c1f55ba328"}
Sep 18 22:30:41.304305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.304207    2551 scope.go:117] "RemoveContainer" containerID="6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"
Sep 18 22:30:41.305547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.305529    2551 generic.go:356] "Generic (PLEG): container finished" podID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" containerID="a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9" exitCode=0
Sep 18 22:30:41.305652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.305573    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"
Sep 18 22:30:41.305652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.305615    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" event={"ID":"a8885dba-2e31-4b13-a57f-9fc5635aff6e","Type":"ContainerDied","Data":"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"}
Sep 18 22:30:41.305775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.305666    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc" event={"ID":"a8885dba-2e31-4b13-a57f-9fc5635aff6e","Type":"ContainerDied","Data":"45ac6b617767bce969749f202bfd7b34c29dbcfbcb0eadfee152e9a83a0e2861"}
Sep 18 22:30:41.313553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.313538    2551 scope.go:117] "RemoveContainer" containerID="6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"
Sep 18 22:30:41.313968 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:41.313907    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf\": container with ID starting with 6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf not found: ID does not exist" containerID="6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"
Sep 18 22:30:41.314152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.313980    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf"} err="failed to get container status \"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf\": rpc error: code = NotFound desc = could not find container \"6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf\": container with ID starting with 6789f7f8f18c3dbe37eb7666a2053f4eb9a8b0d01ad0e4f40c405a4997dfc6bf not found: ID does not exist"
Sep 18 22:30:41.314152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.314021    2551 scope.go:117] "RemoveContainer" containerID="a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"
Sep 18 22:30:41.320368 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.320345    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:30:41.322743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.322726    2551 scope.go:117] "RemoveContainer" containerID="a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"
Sep 18 22:30:41.322989 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:41.322971    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9\": container with ID starting with a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9 not found: ID does not exist" containerID="a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"
Sep 18 22:30:41.323058 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.322994    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9"} err="failed to get container status \"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9\": rpc error: code = NotFound desc = could not find container \"a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9\": container with ID starting with a5d541440f90618493ea253099eb127338dd5c1bcc1bf5db4ed4df467dc9bbe9 not found: ID does not exist"
Sep 18 22:30:41.325577 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.325556    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:41.328921 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.328894    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-67d98638-84b8cbfc74-gpclg"]
Sep 18 22:30:41.337322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.337300    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:41.342070 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.342050    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-migrate-67d98638-6b455b4bb9-4nfbc"]
Sep 18 22:30:41.835065 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835016    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:30:41.835633 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835614    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="80c21ac7-d6c8-48f2-95ec-e04d5548e13e" containerName="revoke-keys"
Sep 18 22:30:41.835732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835634    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="80c21ac7-d6c8-48f2-95ec-e04d5548e13e" containerName="revoke-keys"
Sep 18 22:30:41.835732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835663    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" containerName="maas-api"
Sep 18 22:30:41.835732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835671    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" containerName="maas-api"
Sep 18 22:30:41.835732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835701    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dd69384-92ae-49fb-b1ae-906e917554b3" containerName="payload-pre-processing"
Sep 18 22:30:41.835732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835710    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dd69384-92ae-49fb-b1ae-906e917554b3" containerName="payload-pre-processing"
Sep 18 22:30:41.835955 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835792    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" containerName="maas-api"
Sep 18 22:30:41.835955 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835803    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="80c21ac7-d6c8-48f2-95ec-e04d5548e13e" containerName="revoke-keys"
Sep 18 22:30:41.835955 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.835810    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dd69384-92ae-49fb-b1ae-906e917554b3" containerName="payload-pre-processing"
Sep 18 22:30:41.840671 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.840650    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.843399 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.843383    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-9d0906fb-openshift-default-dockercfg-82dll\""
Sep 18 22:30:41.850333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.850307    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:30:41.907826 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.907925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907837    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4p46\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.907925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907858    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.907925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907905    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.908056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907962    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.908056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.907983    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.908056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.908013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.908154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.908073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:41.908154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:41.908121    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009371    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009482    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009522    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4p46\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009548    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009579    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009632    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009661    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009703    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.009858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.009734    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.010083 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.010055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.010083 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.010072    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.010225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.010156    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.010264 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.010243    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.011804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.011784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.012022 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.012005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.022350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4p46\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.022369    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token\") pod \"e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") " pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.151853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.151828    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:42.290883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.290812    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dd69384-92ae-49fb-b1ae-906e917554b3" path="/var/lib/kubelet/pods/6dd69384-92ae-49fb-b1ae-906e917554b3/volumes"
Sep 18 22:30:42.291186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.291173    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="80c21ac7-d6c8-48f2-95ec-e04d5548e13e" path="/var/lib/kubelet/pods/80c21ac7-d6c8-48f2-95ec-e04d5548e13e/volumes"
Sep 18 22:30:42.291533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.291520    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a8885dba-2e31-4b13-a57f-9fc5635aff6e" path="/var/lib/kubelet/pods/a8885dba-2e31-4b13-a57f-9fc5635aff6e/volumes"
Sep 18 22:30:42.432376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.432352    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:30:42.435177 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:42.435152    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod331fddfe_3cad_4c9d_8c75_cff661535f93.slice/crio-1edee573fcd1834c4939a4b496b9c07f9074640eae2e05d4083bc937f0d479dd WatchSource:0}: Error finding container 1edee573fcd1834c4939a4b496b9c07f9074640eae2e05d4083bc937f0d479dd: Status 404 returned error can't find the container with id 1edee573fcd1834c4939a4b496b9c07f9074640eae2e05d4083bc937f0d479dd
Sep 18 22:30:42.437751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.437720    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:42.437841 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.437794    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:42.437909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:42.437843    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:30:43.320166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:43.320124    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerStarted","Data":"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"}
Sep 18 22:30:43.320603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:43.320244    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerStarted","Data":"1edee573fcd1834c4939a4b496b9c07f9074640eae2e05d4083bc937f0d479dd"}
Sep 18 22:30:43.344362 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:43.344316    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podStartSLOduration=2.344301141 podStartE2EDuration="2.344301141s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:43.341764978 +0000 UTC m=+835.523349549" watchObservedRunningTime="2026-09-18 22:30:43.344301141 +0000 UTC m=+835.525885678"
Sep 18 22:30:44.152699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.152662    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:44.157717 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.157691    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:44.326759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.326724    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:44.327603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.327583    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:30:44.353353 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.353326    2551 patch_prober.go:28] interesting pod/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:30:44.353479 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:44.353373    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:30:46.341469 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.341440    2551 generic.go:356] "Generic (PLEG): container finished" podID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerID="0e139c60d8b787312a06f19b3bd5895ccbdf58236dd9bdcb4d002ae1b9dae5d9" exitCode=0
Sep 18 22:30:46.341865 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.341511    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerDied","Data":"0e139c60d8b787312a06f19b3bd5895ccbdf58236dd9bdcb4d002ae1b9dae5d9"}
Sep 18 22:30:46.341865 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.341575    2551 scope.go:117] "RemoveContainer" containerID="8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d"
Sep 18 22:30:46.361796 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:46.361778    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d\": container with ID starting with 8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d not found: ID does not exist" containerID="8f2f2921fa818aae6545851025db78601e1a1fc79dd02c496164dacb18987c8d"
Sep 18 22:30:46.365077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.365061    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:46.447242 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447204    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447242 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447246    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447267    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447295    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447345    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447399    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qthrb\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.447438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447416    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447496    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447533    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs\") pod \"c4988f2f-9e13-4319-9632-9618ce5371e3\" (UID: \"c4988f2f-9e13-4319-9632-9618ce5371e3\") "
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447602    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447610    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data" (OuterVolumeSpecName: "istio-data") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447808    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447847    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.447996    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.448052    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.448069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.448067    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.448556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.448083    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.448556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.448119    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c4988f2f-9e13-4319-9632-9618ce5371e3-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.449917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.449888    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:30:46.450059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.449893    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token" (OuterVolumeSpecName: "istio-token") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:46.450188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.450169    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb" (OuterVolumeSpecName: "kube-api-access-qthrb") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "kube-api-access-qthrb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:30:46.450245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.450204    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "c4988f2f-9e13-4319-9632-9618ce5371e3" (UID: "c4988f2f-9e13-4319-9632-9618ce5371e3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:30:46.548879 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.548849    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.548879 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.548877    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.548879 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.548886    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qthrb\" (UniqueName: \"kubernetes.io/projected/c4988f2f-9e13-4319-9632-9618ce5371e3-kube-api-access-qthrb\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.549054 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.548896    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c4988f2f-9e13-4319-9632-9618ce5371e3-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:30:46.779531 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.779446    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:30:46.779844 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.779827    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:46.779844 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.779845    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:46.779979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.779918    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:46.784393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.784369    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.786790 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.786762    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-9d0906fb-serving-cert\""
Sep 18 22:30:46.794612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.794589    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:30:46.851392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.851359    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cgxhs\" (UniqueName: \"kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.851522 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.851400    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.851522 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.851419    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.851522 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.851509    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.952696 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.952669    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cgxhs\" (UniqueName: \"kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.952857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.952710    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.952857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.952727    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.952857 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:46.952820    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-serving-cert: secret "maas-api-e2e-ait-9d0906fb-serving-cert" not found
Sep 18 22:30:46.953012 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.952861    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.953012 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:30:46.952869    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls podName:99ebcf8c-53a2-4815-bd6f-9208180b6472 nodeName:}" failed. No retries permitted until 2026-09-18 22:30:47.452854525 +0000 UTC m=+839.634439066 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls") pod "maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" (UID: "99ebcf8c-53a2-4815-bd6f-9208180b6472") : secret "maas-api-e2e-ait-9d0906fb-serving-cert" not found
Sep 18 22:30:46.953435 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.953418    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.955023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.955005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:46.963225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:46.963205    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cgxhs\" (UniqueName: \"kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:47.348445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.348416    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"
Sep 18 22:30:47.348445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.348425    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9" event={"ID":"c4988f2f-9e13-4319-9632-9618ce5371e3","Type":"ContainerDied","Data":"b0d7ddd61f0009d671fded8158a3f6135285f971e8141d5e2f8c4ae764e7e5e8"}
Sep 18 22:30:47.348847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.348479    2551 scope.go:117] "RemoveContainer" containerID="0e139c60d8b787312a06f19b3bd5895ccbdf58236dd9bdcb4d002ae1b9dae5d9"
Sep 18 22:30:47.385840 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.385813    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:30:47.399314 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.399286    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-migrate-67d98638-gw-openshift-default-66d88858d8-kqtv9"]
Sep 18 22:30:47.457425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.457398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:47.459814 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.459789    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") pod \"maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:47.546793 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.546757    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:30:47.547171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.547157    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:47.547171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.547171    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:47.547281 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.547248    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" containerName="istio-proxy"
Sep 18 22:30:47.551783 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.551758    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.557320 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.557296    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-9d0906fb-dockercfg-k6596\""
Sep 18 22:30:47.557910 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.557891    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-9d0906fb\""
Sep 18 22:30:47.558145 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.558127    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.558207 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.558153    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lt8vt\" (UniqueName: \"kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.570269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.570142    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:30:47.659563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.659531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.659563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.659570    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lt8vt\" (UniqueName: \"kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.660322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.660303    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.667918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.667894    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lt8vt\" (UniqueName: \"kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt\") pod \"payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.696703 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.696662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:47.864237 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.864206    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:47.977431 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:47.977405    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:30:47.978846 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:47.978817    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod99ebcf8c_53a2_4815_bd6f_9208180b6472.slice/crio-6b6679c7a5c32c33fbd7e8079127fd0a91328becce1d7600f8d9faf1ccc77d19 WatchSource:0}: Error finding container 6b6679c7a5c32c33fbd7e8079127fd0a91328becce1d7600f8d9faf1ccc77d19: Status 404 returned error can't find the container with id 6b6679c7a5c32c33fbd7e8079127fd0a91328becce1d7600f8d9faf1ccc77d19
Sep 18 22:30:48.142452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.142420    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:30:48.143664 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:30:48.143635    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4dd69674_596c_4d12_9e27_e4f0d99ed87b.slice/crio-1d53e1e693e047bf9cb1e1160a70cfd0e8b4d891363dea962e4958a1a05b0652 WatchSource:0}: Error finding container 1d53e1e693e047bf9cb1e1160a70cfd0e8b4d891363dea962e4958a1a05b0652: Status 404 returned error can't find the container with id 1d53e1e693e047bf9cb1e1160a70cfd0e8b4d891363dea962e4958a1a05b0652
Sep 18 22:30:48.292753 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.292713    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c4988f2f-9e13-4319-9632-9618ce5371e3" path="/var/lib/kubelet/pods/c4988f2f-9e13-4319-9632-9618ce5371e3/volumes"
Sep 18 22:30:48.355928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.355896    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" event={"ID":"4dd69674-596c-4d12-9e27-e4f0d99ed87b","Type":"ContainerStarted","Data":"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"}
Sep 18 22:30:48.356362 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.355956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" event={"ID":"4dd69674-596c-4d12-9e27-e4f0d99ed87b","Type":"ContainerStarted","Data":"1d53e1e693e047bf9cb1e1160a70cfd0e8b4d891363dea962e4958a1a05b0652"}
Sep 18 22:30:48.356362 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.356055    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:30:48.357266 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.357241    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" event={"ID":"99ebcf8c-53a2-4815-bd6f-9208180b6472","Type":"ContainerStarted","Data":"6b6679c7a5c32c33fbd7e8079127fd0a91328becce1d7600f8d9faf1ccc77d19"}
Sep 18 22:30:48.374221 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:48.374173    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" podStartSLOduration=1.3741575799999999 podStartE2EDuration="1.37415758s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:48.371826228 +0000 UTC m=+840.553410774" watchObservedRunningTime="2026-09-18 22:30:48.37415758 +0000 UTC m=+840.555742125"
Sep 18 22:30:49.364234 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:49.364195    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" event={"ID":"99ebcf8c-53a2-4815-bd6f-9208180b6472","Type":"ContainerStarted","Data":"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"}
Sep 18 22:30:49.364629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:49.364393    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:49.384610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:49.384563    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" podStartSLOduration=2.985382778 podStartE2EDuration="3.384532016s" totalImagesPullingTime="399.149238ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:30:46 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:30:49.382858497 +0000 UTC m=+841.564443042" watchObservedRunningTime="2026-09-18 22:30:49.384532016 +0000 UTC m=+841.566116558"
Sep 18 22:30:55.375327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:55.375297    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:30:57.193988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:57.193947    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:30:59.366742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:30:59.366713    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:31:08.468667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.468634    2551 generic.go:356] "Generic (PLEG): container finished" podID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerID="cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a" exitCode=0
Sep 18 22:31:08.469169 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.468725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerDied","Data":"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"}
Sep 18 22:31:08.469241 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.469169    2551 scope.go:117] "RemoveContainer" containerID="cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"
Sep 18 22:31:08.470292 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.470257    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:08.470379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.470348    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:08.470425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:08.470391    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:09.475841 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:09.475806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerStarted","Data":"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"}
Sep 18 22:31:10.152436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:10.152398    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:10.153816 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:10.153782    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 18 22:31:10.153948 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:10.153846    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 18 22:31:11.152556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:11.152515    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 18 22:31:11.153115 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:11.152585    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 18 22:31:12.152995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:12.152958    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 18 22:31:12.153985 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:12.153575    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 18 22:31:13.152732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:13.152684    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 18 22:31:13.152945 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:13.152755    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 18 22:31:14.152787 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:14.152748    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused" start-of-body=
Sep 18 22:31:14.153391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:14.153358    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.48:15021/healthz/ready\": dial tcp 10.132.0.48:15021: connect: connection refused"
Sep 18 22:31:14.327172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:14.327137    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:15.156495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:15.156466    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:15.157483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:15.157465    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:19.121073 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:19.121016    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:31:21.281567 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.281532    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"]
Sep 18 22:31:21.285337 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.285321    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:21.287578 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.287560    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:31:21.293373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.293349    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"]
Sep 18 22:31:21.366077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.366025    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-917b58-jfb97\" (UID: \"a6a98ebb-b86c-416a-8df9-82e74263db32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:21.467066 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.467017    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-917b58-jfb97\" (UID: \"a6a98ebb-b86c-416a-8df9-82e74263db32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:21.467629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.467610    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-917b58-jfb97\" (UID: \"a6a98ebb-b86c-416a-8df9-82e74263db32\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:21.596817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.596741    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:21.869506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:21.869482    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"]
Sep 18 22:31:21.871440 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:31:21.871408    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6a98ebb_b86c_416a_8df9_82e74263db32.slice/crio-8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e WatchSource:0}: Error finding container 8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e: Status 404 returned error can't find the container with id 8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e
Sep 18 22:31:22.549444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:22.549350    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97" event={"ID":"a6a98ebb-b86c-416a-8df9-82e74263db32","Type":"ContainerStarted","Data":"6382b2c87bee100bebd01015e1dd265d3eb3d219f173d498ffb75e06a250baea"}
Sep 18 22:31:22.549444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:22.549407    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97" event={"ID":"a6a98ebb-b86c-416a-8df9-82e74263db32","Type":"ContainerStarted","Data":"8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e"}
Sep 18 22:31:22.564885 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:22.564835    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97" podStartSLOduration=1.564818804 podStartE2EDuration="1.564818804s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:31:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:31:22.563451971 +0000 UTC m=+874.745036518" watchObservedRunningTime="2026-09-18 22:31:22.564818804 +0000 UTC m=+874.746403350"
Sep 18 22:31:23.555713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:23.555678    2551 generic.go:356] "Generic (PLEG): container finished" podID="a6a98ebb-b86c-416a-8df9-82e74263db32" containerID="6382b2c87bee100bebd01015e1dd265d3eb3d219f173d498ffb75e06a250baea" exitCode=0
Sep 18 22:31:23.556095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:23.555765    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97" event={"ID":"a6a98ebb-b86c-416a-8df9-82e74263db32","Type":"ContainerDied","Data":"6382b2c87bee100bebd01015e1dd265d3eb3d219f173d498ffb75e06a250baea"}
Sep 18 22:31:24.694804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:24.694779    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:24.798395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:24.798357    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca\") pod \"a6a98ebb-b86c-416a-8df9-82e74263db32\" (UID: \"a6a98ebb-b86c-416a-8df9-82e74263db32\") "
Sep 18 22:31:24.798727 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:24.798704    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "a6a98ebb-b86c-416a-8df9-82e74263db32" (UID: "a6a98ebb-b86c-416a-8df9-82e74263db32"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:31:24.899738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:24.899709    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6a98ebb-b86c-416a-8df9-82e74263db32-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:25.568999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:25.568972    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"
Sep 18 22:31:25.569244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:25.568971    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97" event={"ID":"a6a98ebb-b86c-416a-8df9-82e74263db32","Type":"ContainerDied","Data":"8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e"}
Sep 18 22:31:25.569244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:25.569119    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8dede280e57fe480eb25fe32c291c7fe95563c593e73d0d20bd73abea798b45e"
Sep 18 22:31:27.162935 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.162901    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:31:27.163414 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.163168    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" podUID="99ebcf8c-53a2-4815-bd6f-9208180b6472" containerName="maas-api" containerID="cri-o://4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406" gracePeriod=30
Sep 18 22:31:27.232904 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.232784    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:31:27.234306 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.233552    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" podUID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" containerName="payload-pre-processing" containerID="cri-o://89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233" gracePeriod=30
Sep 18 22:31:27.484634 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.484607    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:31:27.511357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.511332    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:31:27.525752 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525724    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca\") pod \"99ebcf8c-53a2-4815-bd6f-9208180b6472\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") "
Sep 18 22:31:27.525878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525797    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cgxhs\" (UniqueName: \"kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs\") pod \"99ebcf8c-53a2-4815-bd6f-9208180b6472\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") "
Sep 18 22:31:27.525878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525830    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume\") pod \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") "
Sep 18 22:31:27.525976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525880    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs\") pod \"99ebcf8c-53a2-4815-bd6f-9208180b6472\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") "
Sep 18 22:31:27.525976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525909    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-lt8vt\" (UniqueName: \"kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt\") pod \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\" (UID: \"4dd69674-596c-4d12-9e27-e4f0d99ed87b\") "
Sep 18 22:31:27.525976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.525949    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") pod \"99ebcf8c-53a2-4815-bd6f-9208180b6472\" (UID: \"99ebcf8c-53a2-4815-bd6f-9208180b6472\") "
Sep 18 22:31:27.526212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.526063    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "99ebcf8c-53a2-4815-bd6f-9208180b6472" (UID: "99ebcf8c-53a2-4815-bd6f-9208180b6472"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:31:27.526334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.526311    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/99ebcf8c-53a2-4815-bd6f-9208180b6472-openshift-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:27.526589 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.526562    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4dd69674-596c-4d12-9e27-e4f0d99ed87b" (UID: "4dd69674-596c-4d12-9e27-e4f0d99ed87b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:31:27.528774 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.528474    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs" (OuterVolumeSpecName: "kube-api-access-cgxhs") pod "99ebcf8c-53a2-4815-bd6f-9208180b6472" (UID: "99ebcf8c-53a2-4815-bd6f-9208180b6472"). InnerVolumeSpecName "kube-api-access-cgxhs". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:31:27.529056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.528997    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "99ebcf8c-53a2-4815-bd6f-9208180b6472" (UID: "99ebcf8c-53a2-4815-bd6f-9208180b6472"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:31:27.529473 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.529024    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt" (OuterVolumeSpecName: "kube-api-access-lt8vt") pod "4dd69674-596c-4d12-9e27-e4f0d99ed87b" (UID: "4dd69674-596c-4d12-9e27-e4f0d99ed87b"). InnerVolumeSpecName "kube-api-access-lt8vt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:31:27.529473 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.529196    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "99ebcf8c-53a2-4815-bd6f-9208180b6472" (UID: "99ebcf8c-53a2-4815-bd6f-9208180b6472"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:31:27.582203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.582169    2551 generic.go:356] "Generic (PLEG): container finished" podID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" containerID="89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233" exitCode=0
Sep 18 22:31:27.582324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.582207    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"
Sep 18 22:31:27.582324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.582246    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" event={"ID":"4dd69674-596c-4d12-9e27-e4f0d99ed87b","Type":"ContainerDied","Data":"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"}
Sep 18 22:31:27.582324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.582294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr" event={"ID":"4dd69674-596c-4d12-9e27-e4f0d99ed87b","Type":"ContainerDied","Data":"1d53e1e693e047bf9cb1e1160a70cfd0e8b4d891363dea962e4958a1a05b0652"}
Sep 18 22:31:27.582324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.582314    2551 scope.go:117] "RemoveContainer" containerID="89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"
Sep 18 22:31:27.583883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.583860    2551 generic.go:356] "Generic (PLEG): container finished" podID="99ebcf8c-53a2-4815-bd6f-9208180b6472" containerID="4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406" exitCode=0
Sep 18 22:31:27.583970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.583900    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"
Sep 18 22:31:27.583970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.583917    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" event={"ID":"99ebcf8c-53a2-4815-bd6f-9208180b6472","Type":"ContainerDied","Data":"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"}
Sep 18 22:31:27.583970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.583959    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c" event={"ID":"99ebcf8c-53a2-4815-bd6f-9208180b6472","Type":"ContainerDied","Data":"6b6679c7a5c32c33fbd7e8079127fd0a91328becce1d7600f8d9faf1ccc77d19"}
Sep 18 22:31:27.591074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.591044    2551 scope.go:117] "RemoveContainer" containerID="89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"
Sep 18 22:31:27.591417 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:27.591377    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233\": container with ID starting with 89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233 not found: ID does not exist" containerID="89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"
Sep 18 22:31:27.591519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.591422    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233"} err="failed to get container status \"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233\": rpc error: code = NotFound desc = could not find container \"89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233\": container with ID starting with 89c897939f032064c2ce969c766cfad21beec08ad7e01ff33da7cb6d6252e233 not found: ID does not exist"
Sep 18 22:31:27.591519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.591446    2551 scope.go:117] "RemoveContainer" containerID="4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"
Sep 18 22:31:27.599464 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.599447    2551 scope.go:117] "RemoveContainer" containerID="4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"
Sep 18 22:31:27.599730 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:27.599708    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406\": container with ID starting with 4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406 not found: ID does not exist" containerID="4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"
Sep 18 22:31:27.599782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.599737    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406"} err="failed to get container status \"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406\": rpc error: code = NotFound desc = could not find container \"4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406\": container with ID starting with 4d46ec4abf8b0ac30b9ac803174356e0a6da013589d0538fe2dcab0f9f6f5406 not found: ID does not exist"
Sep 18 22:31:27.606197 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.606175    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:31:27.610000 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.609980    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-9d0906fb-67568f79f8-jbhnr"]
Sep 18 22:31:27.620129 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.620109    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:31:27.626374 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626351    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-9d0906fb-59b4885677-9fl6c"]
Sep 18 22:31:27.626722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626708    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-maas-api-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:27.626772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626725    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cgxhs\" (UniqueName: \"kubernetes.io/projected/99ebcf8c-53a2-4815-bd6f-9208180b6472-kube-api-access-cgxhs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:27.626772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626734    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4dd69674-596c-4d12-9e27-e4f0d99ed87b-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:27.626772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626744    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/99ebcf8c-53a2-4815-bd6f-9208180b6472-metrics-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:27.626772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:27.626752    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lt8vt\" (UniqueName: \"kubernetes.io/projected/4dd69674-596c-4d12-9e27-e4f0d99ed87b-kube-api-access-lt8vt\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:28.290852 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:28.290822    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" path="/var/lib/kubelet/pods/4dd69674-596c-4d12-9e27-e4f0d99ed87b/volumes"
Sep 18 22:31:28.291274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:28.291256    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="99ebcf8c-53a2-4815-bd6f-9208180b6472" path="/var/lib/kubelet/pods/99ebcf8c-53a2-4815-bd6f-9208180b6472/volumes"
Sep 18 22:31:28.522515 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:28.522436    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:31:28.522714 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:28.522686    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" containerID="cri-o://e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803" gracePeriod=30
Sep 18 22:31:29.278762 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.278731    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:31:29.279141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279128    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" containerName="payload-pre-processing"
Sep 18 22:31:29.279191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279143    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" containerName="payload-pre-processing"
Sep 18 22:31:29.279191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279157    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="99ebcf8c-53a2-4815-bd6f-9208180b6472" containerName="maas-api"
Sep 18 22:31:29.279191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279163    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="99ebcf8c-53a2-4815-bd6f-9208180b6472" containerName="maas-api"
Sep 18 22:31:29.279191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279184    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a6a98ebb-b86c-416a-8df9-82e74263db32" containerName="revoke-keys"
Sep 18 22:31:29.279191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279190    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a6a98ebb-b86c-416a-8df9-82e74263db32" containerName="revoke-keys"
Sep 18 22:31:29.279347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279244    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a6a98ebb-b86c-416a-8df9-82e74263db32" containerName="revoke-keys"
Sep 18 22:31:29.279347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279258    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4dd69674-596c-4d12-9e27-e4f0d99ed87b" containerName="payload-pre-processing"
Sep 18 22:31:29.279347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.279263    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="99ebcf8c-53a2-4815-bd6f-9208180b6472" containerName="maas-api"
Sep 18 22:31:29.283862 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.283844    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.286303 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.286285    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-49880939-openshift-default-dockercfg-8bl72\""
Sep 18 22:31:29.299078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.299057    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:31:29.340078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340053    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340108    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340158    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340191    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340215    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dg5kd\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340308    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.340416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.340379    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.445689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.445654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.445978    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446021    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446152 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446121    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dg5kd\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446169    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446205    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446254    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446671    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446795 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446700    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.446941 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446916    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.447124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.446985    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.447355 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.447331    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.449281 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.449263    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.449583 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.449562    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.456394 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.456371    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dg5kd\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.460582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.460561    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token\") pod \"e2e-derive-49880939-openshift-default-548d944c4d-dzh8g\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") " pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.594282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.594207    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:29.872131 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.872108    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:31:29.873955 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:31:29.873928    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9cb4e4bd_b8c6_4d72_bd59_0ab526f58f7e.slice/crio-56a5a5d64a0429f661f0bf599a21866e3456ce84a1724ece6175ac3b8b7b1435 WatchSource:0}: Error finding container 56a5a5d64a0429f661f0bf599a21866e3456ce84a1724ece6175ac3b8b7b1435: Status 404 returned error can't find the container with id 56a5a5d64a0429f661f0bf599a21866e3456ce84a1724ece6175ac3b8b7b1435
Sep 18 22:31:29.876138 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.876101    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:29.876211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.876168    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:29.876211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:29.876197    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:30.159440 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:30.159401    2551 patch_prober.go:28] interesting pod/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:31:30.159603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:30.159465    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:31:30.603392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:30.603311    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerStarted","Data":"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"}
Sep 18 22:31:30.603392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:30.603366    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerStarted","Data":"56a5a5d64a0429f661f0bf599a21866e3456ce84a1724ece6175ac3b8b7b1435"}
Sep 18 22:31:30.626952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:30.626906    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podStartSLOduration=1.626892038 podStartE2EDuration="1.626892038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:31:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:31:30.625024261 +0000 UTC m=+882.806608811" watchObservedRunningTime="2026-09-18 22:31:30.626892038 +0000 UTC m=+882.808476583"
Sep 18 22:31:31.595250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:31.595222    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:31.600423 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:31.600398    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:31.608931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:31.608911    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:31.609561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:31.609542    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:31:33.769817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.769793    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:33.784150 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784097    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784152    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784175    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784203    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784239    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784264    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784332    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784362    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n4p46\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784385    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token\") pod \"331fddfe-3cad-4c9d-8c75-cff661535f93\" (UID: \"331fddfe-3cad-4c9d-8c75-cff661535f93\") "
Sep 18 22:31:33.784465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784405    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:31:33.784722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784484    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:31:33.784722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784504    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:31:33.784722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784611    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data" (OuterVolumeSpecName: "istio-data") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:31:33.784928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784789    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.784928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784808    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.784928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784822    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.784928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.784835    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.785211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.785191    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:31:33.786550 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.786529    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:31:33.786641 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.786623    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:31:33.786881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.786854    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46" (OuterVolumeSpecName: "kube-api-access-n4p46") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "kube-api-access-n4p46". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:31:33.786995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.786973    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token" (OuterVolumeSpecName: "istio-token") pod "331fddfe-3cad-4c9d-8c75-cff661535f93" (UID: "331fddfe-3cad-4c9d-8c75-cff661535f93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:31:33.886323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.886291    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.886323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.886318    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.886501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.886330    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/331fddfe-3cad-4c9d-8c75-cff661535f93-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.886501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.886342    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4p46\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-kube-api-access-n4p46\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:33.886501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:33.886355    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/331fddfe-3cad-4c9d-8c75-cff661535f93-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:31:34.626365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.626276    2551 generic.go:356] "Generic (PLEG): container finished" podID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerID="e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803" exitCode=0
Sep 18 22:31:34.626365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.626337    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"
Sep 18 22:31:34.626560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.626399    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerDied","Data":"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"}
Sep 18 22:31:34.626560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.626449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf" event={"ID":"331fddfe-3cad-4c9d-8c75-cff661535f93","Type":"ContainerDied","Data":"1edee573fcd1834c4939a4b496b9c07f9074640eae2e05d4083bc937f0d479dd"}
Sep 18 22:31:34.626560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.626473    2551 scope.go:117] "RemoveContainer" containerID="e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"
Sep 18 22:31:34.660009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.659870    2551 scope.go:117] "RemoveContainer" containerID="cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"
Sep 18 22:31:34.669888 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.669868    2551 scope.go:117] "RemoveContainer" containerID="e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"
Sep 18 22:31:34.670190 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:34.670171    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803\": container with ID starting with e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803 not found: ID does not exist" containerID="e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"
Sep 18 22:31:34.670267 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.670197    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803"} err="failed to get container status \"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803\": rpc error: code = NotFound desc = could not find container \"e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803\": container with ID starting with e22956423d4fedc533acfbac989f3d1730b2277e51d229adbf91676f12ab4803 not found: ID does not exist"
Sep 18 22:31:34.670267 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.670215    2551 scope.go:117] "RemoveContainer" containerID="cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"
Sep 18 22:31:34.670491 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:34.670475    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a\": container with ID starting with cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a not found: ID does not exist" containerID="cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"
Sep 18 22:31:34.670536 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.670494    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a"} err="failed to get container status \"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a\": rpc error: code = NotFound desc = could not find container \"cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a\": container with ID starting with cc9a68b83c5ae3114958a1d81aedae19cac4c1360dec357a64c984acb114a91a not found: ID does not exist"
Sep 18 22:31:34.674878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.674856    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:31:34.680691 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:34.680670    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-9d0906fb-openshift-default-549dc59cd4-dm6lf"]
Sep 18 22:31:36.063327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.063295    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"]
Sep 18 22:31:36.065985 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.065958    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-autores-917b58-29829510-4ntx4"]
Sep 18 22:31:36.292111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.292079    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" path="/var/lib/kubelet/pods/331fddfe-3cad-4c9d-8c75-cff661535f93/volumes"
Sep 18 22:31:36.292474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.292453    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92afe595-570c-4df2-ac32-562c2e0bc798" path="/var/lib/kubelet/pods/92afe595-570c-4df2-ac32-562c2e0bc798/volumes"
Sep 18 22:31:36.402674 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.402643    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"]
Sep 18 22:31:36.409807 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.409751    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:31:36.410416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.410394    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:31:36.410416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.410417    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:31:36.410576 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.410564    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:31:36.410639 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.410585    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:31:36.415388 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.415369    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-917b58-jfb97"]
Sep 18 22:31:36.415466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.415455    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.418654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.418635    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-derive-49880939-serving-cert\""
Sep 18 22:31:36.428333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.428313    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:31:36.511986 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.511957    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.512143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.512012    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.512143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.512070    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vp8xl\" (UniqueName: \"kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.512211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.512134    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.613016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.612986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.613185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.613051    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.613185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.613085    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vp8xl\" (UniqueName: \"kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.613185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.613124    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.613341 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:36.613233    2551 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-derive-49880939-serving-cert: secret "maas-api-e2e-derive-49880939-serving-cert" not found
Sep 18 22:31:36.613341 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:31:36.613302    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls podName:89644aeb-37f7-4694-85cf-7b0ec561c367 nodeName:}" failed. No retries permitted until 2026-09-18 22:31:37.113284088 +0000 UTC m=+889.294868616 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls") pod "maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" (UID: "89644aeb-37f7-4694-85cf-7b0ec561c367") : secret "maas-api-e2e-derive-49880939-serving-cert" not found
Sep 18 22:31:36.613730 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.613711    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.615229 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.615210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:36.623336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:36.623315    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vp8xl\" (UniqueName: \"kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:37.116616 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:37.116566    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:37.118919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:37.118899    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") pod \"maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") " pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:37.328738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:37.328705    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:37.610202 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:37.610180    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:31:37.611734 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:31:37.611713    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod89644aeb_37f7_4694_85cf_7b0ec561c367.slice/crio-93f234d68e5f1a98348d3fe5c78aaf7336d366334f33aadd776b4f222cd88f33 WatchSource:0}: Error finding container 93f234d68e5f1a98348d3fe5c78aaf7336d366334f33aadd776b4f222cd88f33: Status 404 returned error can't find the container with id 93f234d68e5f1a98348d3fe5c78aaf7336d366334f33aadd776b4f222cd88f33
Sep 18 22:31:37.647231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:37.647204    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" event={"ID":"89644aeb-37f7-4694-85cf-7b0ec561c367","Type":"ContainerStarted","Data":"93f234d68e5f1a98348d3fe5c78aaf7336d366334f33aadd776b4f222cd88f33"}
Sep 18 22:31:38.291381 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:38.291310    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a6a98ebb-b86c-416a-8df9-82e74263db32" path="/var/lib/kubelet/pods/a6a98ebb-b86c-416a-8df9-82e74263db32/volumes"
Sep 18 22:31:38.660916 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:38.660877    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" event={"ID":"89644aeb-37f7-4694-85cf-7b0ec561c367","Type":"ContainerStarted","Data":"fd47af164980857be911a060f8bfcdd27727345c13fa5ab2ca8e1d246346bcda"}
Sep 18 22:31:38.661101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:38.660978    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:38.682429 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:38.682382    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" podStartSLOduration=2.341601432 podStartE2EDuration="2.682368197s" totalImagesPullingTime="340.766765ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:31:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:31:38.679868501 +0000 UTC m=+890.861453039" watchObservedRunningTime="2026-09-18 22:31:38.682368197 +0000 UTC m=+890.863952744"
Sep 18 22:31:44.672510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:44.672439    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:31:47.896978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:47.896944    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:31:48.272782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:48.272702    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:31:48.273086 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:48.273069    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:31:58.781365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.781339    2551 generic.go:356] "Generic (PLEG): container finished" podID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerID="c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589" exitCode=0
Sep 18 22:31:58.781775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.781413    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerDied","Data":"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"}
Sep 18 22:31:58.781840 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.781808    2551 scope.go:117] "RemoveContainer" containerID="c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"
Sep 18 22:31:58.782860 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.782829    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:58.782970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.782948    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:58.783059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:58.782999    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:31:59.788044 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:31:59.788000    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerStarted","Data":"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"}
Sep 18 22:32:00.595422 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:00.595385    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:00.596532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:00.596500    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 22:32:00.596661 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:00.596581    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 22:32:01.595099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:01.595066    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 22:32:01.595438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:01.595123    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 22:32:01.609348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:01.609322    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:02.594896 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:02.594856    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 22:32:02.595097 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:02.594922    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 22:32:03.594535 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:03.594502    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 22:32:03.594893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:03.594566    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 22:32:04.595547 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:04.595511    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused" start-of-body=
Sep 18 22:32:04.595919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:04.595578    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.52:15021/healthz/ready\": dial tcp 10.132.0.52:15021: connect: connection refused"
Sep 18 22:32:05.598884 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:05.598855    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:05.599697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:05.599677    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:15.242833 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.242798    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:32:15.243314 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.243221    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:32:15.243314 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.243233    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="331fddfe-3cad-4c9d-8c75-cff661535f93" containerName="istio-proxy"
Sep 18 22:32:15.246219 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.246204    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.248859 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.248837    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"premium-w0-c5fdd3-kserve-self-signed-certs\""
Sep 18 22:32:15.256065 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.256023    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:32:15.276387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gwcst\" (UniqueName: \"kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.276485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276423    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.276485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.276485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276463    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.276602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276495    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.276602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.276528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377229    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377282    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gwcst\" (UniqueName: \"kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377333    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377379 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377371    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377394    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377644 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377623    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377706 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377687    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.377868 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.377847    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.379564 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.379541    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.379819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.379804    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.385289 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.385270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gwcst\" (UniqueName: \"kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst\") pod \"premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") " pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.556090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.555986    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:15.835284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.835256    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:32:15.837391 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:32:15.837362    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70986b94_5bd0_4915_82ab_146327e59ba8.slice/crio-fff1eca6ff21722c0cd4a400bb908af8dbfe9a0c51b4fcf5a5512cfdb7a90abc WatchSource:0}: Error finding container fff1eca6ff21722c0cd4a400bb908af8dbfe9a0c51b4fcf5a5512cfdb7a90abc: Status 404 returned error can't find the container with id fff1eca6ff21722c0cd4a400bb908af8dbfe9a0c51b4fcf5a5512cfdb7a90abc
Sep 18 22:32:15.878559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:15.878534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" event={"ID":"70986b94-5bd0-4915-82ab-146327e59ba8","Type":"ContainerStarted","Data":"fff1eca6ff21722c0cd4a400bb908af8dbfe9a0c51b4fcf5a5512cfdb7a90abc"}
Sep 18 22:32:16.885685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:16.885652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" event={"ID":"70986b94-5bd0-4915-82ab-146327e59ba8","Type":"ContainerStarted","Data":"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"}
Sep 18 22:32:16.902232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:16.902186    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" podStartSLOduration=1.902169399 podStartE2EDuration="1.902169399s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:16.901533078 +0000 UTC m=+929.083117625" watchObservedRunningTime="2026-09-18 22:32:16.902169399 +0000 UTC m=+929.083753945"
Sep 18 22:32:17.763272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:17.763237    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:32:17.763505 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:17.763473    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" podUID="89644aeb-37f7-4694-85cf-7b0ec561c367" containerName="maas-api" containerID="cri-o://fd47af164980857be911a060f8bfcdd27727345c13fa5ab2ca8e1d246346bcda" gracePeriod=30
Sep 18 22:32:17.898795 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:17.898757    2551 generic.go:356] "Generic (PLEG): container finished" podID="89644aeb-37f7-4694-85cf-7b0ec561c367" containerID="fd47af164980857be911a060f8bfcdd27727345c13fa5ab2ca8e1d246346bcda" exitCode=0
Sep 18 22:32:17.899186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:17.898830    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" event={"ID":"89644aeb-37f7-4694-85cf-7b0ec561c367","Type":"ContainerDied","Data":"fd47af164980857be911a060f8bfcdd27727345c13fa5ab2ca8e1d246346bcda"}
Sep 18 22:32:18.019297 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.019235    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:32:18.103221 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.103187    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs\") pod \"89644aeb-37f7-4694-85cf-7b0ec561c367\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") "
Sep 18 22:32:18.103496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.103463    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vp8xl\" (UniqueName: \"kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl\") pod \"89644aeb-37f7-4694-85cf-7b0ec561c367\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") "
Sep 18 22:32:18.103617 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.103577    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") pod \"89644aeb-37f7-4694-85cf-7b0ec561c367\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") "
Sep 18 22:32:18.103677 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.103621    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca\") pod \"89644aeb-37f7-4694-85cf-7b0ec561c367\" (UID: \"89644aeb-37f7-4694-85cf-7b0ec561c367\") "
Sep 18 22:32:18.104117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.104083    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "89644aeb-37f7-4694-85cf-7b0ec561c367" (UID: "89644aeb-37f7-4694-85cf-7b0ec561c367"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:32:18.107128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.106954    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "89644aeb-37f7-4694-85cf-7b0ec561c367" (UID: "89644aeb-37f7-4694-85cf-7b0ec561c367"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:32:18.107425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.107372    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "89644aeb-37f7-4694-85cf-7b0ec561c367" (UID: "89644aeb-37f7-4694-85cf-7b0ec561c367"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:32:18.107425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.107400    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl" (OuterVolumeSpecName: "kube-api-access-vp8xl") pod "89644aeb-37f7-4694-85cf-7b0ec561c367" (UID: "89644aeb-37f7-4694-85cf-7b0ec561c367"). InnerVolumeSpecName "kube-api-access-vp8xl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:32:18.135069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.135009    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:32:18.135337 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.135305    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" containerID="cri-o://7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa" gracePeriod=30
Sep 18 22:32:18.204510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.204478    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-maas-api-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:18.204510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.204504    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/89644aeb-37f7-4694-85cf-7b0ec561c367-openshift-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:18.204510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.204520    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/89644aeb-37f7-4694-85cf-7b0ec561c367-metrics-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:18.204725 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.204530    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vp8xl\" (UniqueName: \"kubernetes.io/projected/89644aeb-37f7-4694-85cf-7b0ec561c367-kube-api-access-vp8xl\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:18.904889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.904856    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"
Sep 18 22:32:18.905322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.904857    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf" event={"ID":"89644aeb-37f7-4694-85cf-7b0ec561c367","Type":"ContainerDied","Data":"93f234d68e5f1a98348d3fe5c78aaf7336d366334f33aadd776b4f222cd88f33"}
Sep 18 22:32:18.905322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.904993    2551 scope.go:117] "RemoveContainer" containerID="fd47af164980857be911a060f8bfcdd27727345c13fa5ab2ca8e1d246346bcda"
Sep 18 22:32:18.946016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.945991    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:32:18.958311 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:18.958285    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-derive-49880939-7c65bc87b8-qwtvf"]
Sep 18 22:32:20.290996 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:20.290963    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89644aeb-37f7-4694-85cf-7b0ec561c367" path="/var/lib/kubelet/pods/89644aeb-37f7-4694-85cf-7b0ec561c367/volumes"
Sep 18 22:32:20.601832 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:20.601738    2551 patch_prober.go:28] interesting pod/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:32:20.601832 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:20.601808    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:32:22.116219 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.116179    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:32:22.116694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.116572    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="89644aeb-37f7-4694-85cf-7b0ec561c367" containerName="maas-api"
Sep 18 22:32:22.116694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.116584    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="89644aeb-37f7-4694-85cf-7b0ec561c367" containerName="maas-api"
Sep 18 22:32:22.116694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.116643    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="89644aeb-37f7-4694-85cf-7b0ec561c367" containerName="maas-api"
Sep 18 22:32:22.121188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.121167    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.123680 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.123659    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-explicit-292146-openshift-default-dockercfg-cwv6s\""
Sep 18 22:32:22.137897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.137867    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:32:22.240575 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240587    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240625    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240666    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240683    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-blhhq\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.240731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.241021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240742    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.241021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.240758    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.341902 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.341868    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.341902 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.341897    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.341960    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.341997    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342094 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342074    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342116    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342144    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342190    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-blhhq\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342226    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342386    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342463    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.342929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.342900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.344453 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.344435    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.344665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.344648    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.350078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.350050    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.350248 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.350231    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-blhhq\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq\") pod \"e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") " pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.435118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.435099    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:22.719524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.719494    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:32:22.721441 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:32:22.721404    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod557cf4ce_39dd_4cc2_8307_f22a31709c38.slice/crio-50d3a8aebb40473a17d7ca3f1c527f601a5f84821c42d21a91617804a117db06 WatchSource:0}: Error finding container 50d3a8aebb40473a17d7ca3f1c527f601a5f84821c42d21a91617804a117db06: Status 404 returned error can't find the container with id 50d3a8aebb40473a17d7ca3f1c527f601a5f84821c42d21a91617804a117db06
Sep 18 22:32:22.723560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.723524    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:22.723662 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.723601    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:22.723662 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.723647    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:22.794933 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.794910    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:32:22.928825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.928786    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerStarted","Data":"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"}
Sep 18 22:32:22.929015 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.928848    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerStarted","Data":"50d3a8aebb40473a17d7ca3f1c527f601a5f84821c42d21a91617804a117db06"}
Sep 18 22:32:22.952259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:22.952212    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podStartSLOduration=0.95219643 podStartE2EDuration="952.19643ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:22.950011518 +0000 UTC m=+935.131596088" watchObservedRunningTime="2026-09-18 22:32:22.95219643 +0000 UTC m=+935.133781025"
Sep 18 22:32:23.391997 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.391972    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:23.435874 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.435802    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:23.441988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.441957    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:23.453682 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453661    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.453792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453706    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.453792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453749    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.453907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453815    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.453907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453845    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.453907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453885    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.454085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453896    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:23.454085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453952    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.454085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.453997    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.454085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454022    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dg5kd\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd\") pod \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\" (UID: \"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e\") "
Sep 18 22:32:23.454273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454088    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data" (OuterVolumeSpecName: "istio-data") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:23.454273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454080    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:23.454387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454362    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:23.454447 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454385    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.454447 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454403    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.454447 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454406    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:32:23.454447 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.454420    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.456246 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.456218    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:32:23.456352 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.456253    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd" (OuterVolumeSpecName: "kube-api-access-dg5kd") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "kube-api-access-dg5kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:32:23.456420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.456379    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token" (OuterVolumeSpecName: "istio-token") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:32:23.456420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.456389    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" (UID: "9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:32:23.555001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.554974    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.555001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.554996    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.555001 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.555005    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.555236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.555014    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.555236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.555023    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.555236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.555055    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dg5kd\" (UniqueName: \"kubernetes.io/projected/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e-kube-api-access-dg5kd\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:32:23.936345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936305    2551 generic.go:356] "Generic (PLEG): container finished" podID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerID="7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa" exitCode=0
Sep 18 22:32:23.936508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936352    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"
Sep 18 22:32:23.936508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936383    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerDied","Data":"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"}
Sep 18 22:32:23.936508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936440    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g" event={"ID":"9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e","Type":"ContainerDied","Data":"56a5a5d64a0429f661f0bf599a21866e3456ce84a1724ece6175ac3b8b7b1435"}
Sep 18 22:32:23.936508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936471    2551 scope.go:117] "RemoveContainer" containerID="7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"
Sep 18 22:32:23.936966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.936944    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:23.938232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.938211    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:23.947084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.947067    2551 scope.go:117] "RemoveContainer" containerID="c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"
Sep 18 22:32:23.954883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.954868    2551 scope.go:117] "RemoveContainer" containerID="7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"
Sep 18 22:32:23.955160 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:32:23.955143    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa\": container with ID starting with 7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa not found: ID does not exist" containerID="7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"
Sep 18 22:32:23.955231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.955168    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa"} err="failed to get container status \"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa\": rpc error: code = NotFound desc = could not find container \"7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa\": container with ID starting with 7be3217d4431bdb542d118d6d2182f6336b0edc4f3b795add09f9fe4584479aa not found: ID does not exist"
Sep 18 22:32:23.955231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.955184    2551 scope.go:117] "RemoveContainer" containerID="c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"
Sep 18 22:32:23.955385 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:32:23.955371    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589\": container with ID starting with c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589 not found: ID does not exist" containerID="c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"
Sep 18 22:32:23.955424 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.955389    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589"} err="failed to get container status \"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589\": rpc error: code = NotFound desc = could not find container \"c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589\": container with ID starting with c593aa05ef10c9deb2f1e25c58bf0518f45cb4398d254824198272da229cb589 not found: ID does not exist"
Sep 18 22:32:23.978626 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.978599    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:32:23.982703 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:23.982679    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-49880939-openshift-default-548d944c4d-dzh8g"]
Sep 18 22:32:24.291356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:24.291275    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" path="/var/lib/kubelet/pods/9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e/volumes"
Sep 18 22:32:25.556130 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:25.556102    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:25.556493 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:25.556149    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:25.568445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:25.568424    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:25.959536 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:25.959510    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:32:29.901667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.901635    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:32:29.902072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902044    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.902072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902059    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.902160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902079    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.902160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902089    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.902232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902163    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.902232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.902172    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9cb4e4bd-b8c6-4d72-bd59-0ab526f58f7e" containerName="istio-proxy"
Sep 18 22:32:29.906925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.906903    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:29.909322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.909301    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-explicit-292146-dockercfg-plgbb\""
Sep 18 22:32:29.909422 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.909301    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-explicit-292146\""
Sep 18 22:32:29.913963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:29.913941    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:32:30.013411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.013379    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.013582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.013439    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nlj9t\" (UniqueName: \"kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.114855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.114812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.115025 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.114872    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nlj9t\" (UniqueName: \"kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.115661 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.115640    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.128804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.128771    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-nlj9t\" (UniqueName: \"kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t\") pod \"payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") " pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.219707 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.219618    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:30.494606 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.494570    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:32:30.496792 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:32:30.496759    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf3499a34_d2ac_4dbc_ae25_67fde5123af8.slice/crio-22ea00529d7750d5c67f9230b0e052b0c2e130e2f013377bf7009a3c4bdafc4b WatchSource:0}: Error finding container 22ea00529d7750d5c67f9230b0e052b0c2e130e2f013377bf7009a3c4bdafc4b: Status 404 returned error can't find the container with id 22ea00529d7750d5c67f9230b0e052b0c2e130e2f013377bf7009a3c4bdafc4b
Sep 18 22:32:30.986570 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.986538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" event={"ID":"f3499a34-d2ac-4dbc-ae25-67fde5123af8","Type":"ContainerStarted","Data":"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"}
Sep 18 22:32:30.986999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.986586    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" event={"ID":"f3499a34-d2ac-4dbc-ae25-67fde5123af8","Type":"ContainerStarted","Data":"22ea00529d7750d5c67f9230b0e052b0c2e130e2f013377bf7009a3c4bdafc4b"}
Sep 18 22:32:30.986999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:30.986630    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:40.408506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:40.408451    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" podStartSLOduration=11.408435062 podStartE2EDuration="11.408435062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:32:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:32:31.01510139 +0000 UTC m=+943.196685933" watchObservedRunningTime="2026-09-18 22:32:40.408435062 +0000 UTC m=+952.590019607"
Sep 18 22:32:40.409098 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:40.409073    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:32:41.995143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:41.995110    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:32:51.107916 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.107882    2551 generic.go:356] "Generic (PLEG): container finished" podID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerID="0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2" exitCode=0
Sep 18 22:32:51.108483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.107961    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerDied","Data":"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"}
Sep 18 22:32:51.108562 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.108528    2551 scope.go:117] "RemoveContainer" containerID="0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"
Sep 18 22:32:51.112699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.112663    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:51.112825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.112732    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:51.112825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:51.112767    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:32:52.115438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:52.115404    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerStarted","Data":"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"}
Sep 18 22:32:52.436411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:52.436083    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:52.437914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:52.437606    2551 patch_prober.go:28] interesting pod/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 18 22:32:52.437914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:52.437660    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 18 22:32:53.436208 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:53.436165    2551 patch_prober.go:28] interesting pod/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 18 22:32:53.436813 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:53.436780    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 18 22:32:53.937172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:53.937125    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:54.436438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:54.436399    2551 patch_prober.go:28] interesting pod/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 18 22:32:54.436968 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:54.436467    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 18 22:32:55.436177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:55.436140    2551 patch_prober.go:28] interesting pod/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 18 22:32:55.436470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:55.436438    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 18 22:32:56.435602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:56.435559    2551 patch_prober.go:28] interesting pod/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused" start-of-body=
Sep 18 22:32:56.435920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:56.435888    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.55:15021/healthz/ready\": dial tcp 10.132.0.55:15021: connect: connection refused"
Sep 18 22:32:57.440416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:57.440383    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:57.441621 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:57.441599    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:32:58.900432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:32:58.900397    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:33:16.403971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:16.403902    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:33:43.006546 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:43.006510    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:33:49.757285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.757251    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:33:49.760778 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.760759    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.763262 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.763245    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-292146\"/\"default-dockercfg-xbzn9\""
Sep 18 22:33:49.764244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.764226    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-292146\"/\"kube-root-ca.crt\""
Sep 18 22:33:49.764337 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.764319    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-explicit-292146\"/\"explicit-model-292146-kserve-self-signed-certs\""
Sep 18 22:33:49.764385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.764370    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-explicit-292146\"/\"openshift-service-ca.crt\""
Sep 18 22:33:49.771214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.771192    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:33:49.814223 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814195    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.814333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814230    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.814333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kmr94\" (UniqueName: \"kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.814439 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814373    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.814485 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814449    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.814527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.814496    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915229 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915208    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915254    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915303    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915327 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915319    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915352    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kmr94\" (UniqueName: \"kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915653 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915629    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915755 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915678    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.915755 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.915746    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.917479 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.917461    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.917745 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.917724    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:49.924918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:49.924896    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kmr94\" (UniqueName: \"kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94\") pod \"explicit-model-292146-kserve-7c984cc5df-ljtrs\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") " pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:50.070637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:50.070573    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:33:50.368696 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:33:50.368634    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6f2b38e7_7ae9_4c99_9931_7933ec81f8c0.slice/crio-83480d4c051ab3d543f635d3f65fe643cbb8bff4c28bf0cf0fdf102ac0ff3610 WatchSource:0}: Error finding container 83480d4c051ab3d543f635d3f65fe643cbb8bff4c28bf0cf0fdf102ac0ff3610: Status 404 returned error can't find the container with id 83480d4c051ab3d543f635d3f65fe643cbb8bff4c28bf0cf0fdf102ac0ff3610
Sep 18 22:33:50.371779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:50.371756    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:33:50.420051 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:50.420006    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" event={"ID":"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0","Type":"ContainerStarted","Data":"83480d4c051ab3d543f635d3f65fe643cbb8bff4c28bf0cf0fdf102ac0ff3610"}
Sep 18 22:33:51.427113 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:51.427081    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" event={"ID":"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0","Type":"ContainerStarted","Data":"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"}
Sep 18 22:33:51.444984 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:33:51.444931    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" podStartSLOduration=2.444916571 podStartE2EDuration="2.444916571s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:33:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:33:51.443229996 +0000 UTC m=+1023.624814544" watchObservedRunningTime="2026-09-18 22:33:51.444916571 +0000 UTC m=+1023.626501117"
Sep 18 22:34:00.071015 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:00.070979    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:00.071616 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:00.071058    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:00.083363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:00.083332    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:00.487914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:00.487879    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:05.211657 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.211622    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:05.215263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.215246    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.217846 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.217819    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-939c7fd8-dockercfg-t8czq\""
Sep 18 22:34:05.218084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.218068    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-939c7fd8\""
Sep 18 22:34:05.226606 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.226585    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:05.247892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.247860    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.248012 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.247896    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lv2n\" (UniqueName: \"kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.348454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.348429    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.348592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.348457    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lv2n\" (UniqueName: \"kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.349232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.349208    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.357087 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.357051    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lv2n\" (UniqueName: \"kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n\") pod \"payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.526147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.526091    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:05.801917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:05.801891    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:05.803651 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:34:05.803628    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60b5104a_cd47_4a20_b9f0_f4085a675163.slice/crio-6295b2558a0315cdb5b1b7127357c684139db9b48eb9442bc5736cb97e20a913 WatchSource:0}: Error finding container 6295b2558a0315cdb5b1b7127357c684139db9b48eb9442bc5736cb97e20a913: Status 404 returned error can't find the container with id 6295b2558a0315cdb5b1b7127357c684139db9b48eb9442bc5736cb97e20a913
Sep 18 22:34:06.510209 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:06.510179    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" event={"ID":"60b5104a-cd47-4a20-b9f0-f4085a675163","Type":"ContainerStarted","Data":"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"}
Sep 18 22:34:06.510608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:06.510222    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" event={"ID":"60b5104a-cd47-4a20-b9f0-f4085a675163","Type":"ContainerStarted","Data":"6295b2558a0315cdb5b1b7127357c684139db9b48eb9442bc5736cb97e20a913"}
Sep 18 22:34:06.510608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:06.510320    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:06.528806 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:06.528764    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" podStartSLOduration=1.528751292 podStartE2EDuration="1.528751292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:06.526358582 +0000 UTC m=+1038.707943127" watchObservedRunningTime="2026-09-18 22:34:06.528751292 +0000 UTC m=+1038.710335837"
Sep 18 22:34:13.226957 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:13.226925    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:34:17.518391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:17.518360    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:22.300211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.300178    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"]
Sep 18 22:34:22.303915 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.303894    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:22.306251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.306234    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:34:22.311918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.311896    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"]
Sep 18 22:34:22.392732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.392703    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-292146-zmdkt\" (UID: \"2ede803a-1bb9-450b-8074-385fcf65cb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:22.494185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.494158    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-292146-zmdkt\" (UID: \"2ede803a-1bb9-450b-8074-385fcf65cb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:22.494737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.494720    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-292146-zmdkt\" (UID: \"2ede803a-1bb9-450b-8074-385fcf65cb82\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:22.615126 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.615058    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:22.889653 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:22.889623    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"]
Sep 18 22:34:22.894491 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:34:22.891720    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ede803a_1bb9_450b_8074_385fcf65cb82.slice/crio-21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44 WatchSource:0}: Error finding container 21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44: Status 404 returned error can't find the container with id 21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44
Sep 18 22:34:23.606561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:23.606466    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt" event={"ID":"2ede803a-1bb9-450b-8074-385fcf65cb82","Type":"ContainerStarted","Data":"20f63baa5fd65b3f3b77f1d4cd6e9f77d136f292d67fa1c3f67da5ae11dc54ee"}
Sep 18 22:34:23.606561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:23.606521    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt" event={"ID":"2ede803a-1bb9-450b-8074-385fcf65cb82","Type":"ContainerStarted","Data":"21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44"}
Sep 18 22:34:23.624890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:23.624843    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt" podStartSLOduration=1.624830958 podStartE2EDuration="1.624830958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:23.623667836 +0000 UTC m=+1055.805252382" watchObservedRunningTime="2026-09-18 22:34:23.624830958 +0000 UTC m=+1055.806415564"
Sep 18 22:34:24.613555 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:24.613523    2551 generic.go:356] "Generic (PLEG): container finished" podID="2ede803a-1bb9-450b-8074-385fcf65cb82" containerID="20f63baa5fd65b3f3b77f1d4cd6e9f77d136f292d67fa1c3f67da5ae11dc54ee" exitCode=0
Sep 18 22:34:24.613953 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:24.613613    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt" event={"ID":"2ede803a-1bb9-450b-8074-385fcf65cb82","Type":"ContainerDied","Data":"20f63baa5fd65b3f3b77f1d4cd6e9f77d136f292d67fa1c3f67da5ae11dc54ee"}
Sep 18 22:34:25.747552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:25.747520    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:34:25.766545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:25.766518    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:25.928259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:25.928229    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca\") pod \"2ede803a-1bb9-450b-8074-385fcf65cb82\" (UID: \"2ede803a-1bb9-450b-8074-385fcf65cb82\") "
Sep 18 22:34:25.928610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:25.928586    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2ede803a-1bb9-450b-8074-385fcf65cb82" (UID: "2ede803a-1bb9-450b-8074-385fcf65cb82"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:26.028906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:26.028879    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2ede803a-1bb9-450b-8074-385fcf65cb82-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:26.625905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:26.625818    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt" event={"ID":"2ede803a-1bb9-450b-8074-385fcf65cb82","Type":"ContainerDied","Data":"21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44"}
Sep 18 22:34:26.625905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:26.625837    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"
Sep 18 22:34:26.625905 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:26.625870    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="21c5e6ee1505d8d2ce705293599a3b9fed2fb5b05c3c7787212c5f2eb7d84f44"
Sep 18 22:34:33.345411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.345377    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:34:33.345823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.345578    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" podUID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" containerName="payload-pre-processing" containerID="cri-o://8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e" gracePeriod=30
Sep 18 22:34:33.423653 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.423616    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:34:33.595082 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.595053    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"]
Sep 18 22:34:33.598384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.598330    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-292146-zmdkt"]
Sep 18 22:34:33.602155 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.602135    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:34:33.667279 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.667249    2551 generic.go:356] "Generic (PLEG): container finished" podID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" containerID="8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e" exitCode=0
Sep 18 22:34:33.667438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.667289    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"
Sep 18 22:34:33.667438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.667298    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" event={"ID":"f3499a34-d2ac-4dbc-ae25-67fde5123af8","Type":"ContainerDied","Data":"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"}
Sep 18 22:34:33.667438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.667368    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9" event={"ID":"f3499a34-d2ac-4dbc-ae25-67fde5123af8","Type":"ContainerDied","Data":"22ea00529d7750d5c67f9230b0e052b0c2e130e2f013377bf7009a3c4bdafc4b"}
Sep 18 22:34:33.667438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.667401    2551 scope.go:117] "RemoveContainer" containerID="8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"
Sep 18 22:34:33.676951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.676926    2551 scope.go:117] "RemoveContainer" containerID="8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"
Sep 18 22:34:33.677233 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:34:33.677213    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e\": container with ID starting with 8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e not found: ID does not exist" containerID="8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"
Sep 18 22:34:33.677313 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.677243    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e"} err="failed to get container status \"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e\": rpc error: code = NotFound desc = could not find container \"8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e\": container with ID starting with 8ad032f85b2b0fe5850c62100ecd404306c54e5bcf72d6fa360d37975fd0cd0e not found: ID does not exist"
Sep 18 22:34:33.700339 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.700316    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume\") pod \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") "
Sep 18 22:34:33.700460 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.700371    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nlj9t\" (UniqueName: \"kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t\") pod \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\" (UID: \"f3499a34-d2ac-4dbc-ae25-67fde5123af8\") "
Sep 18 22:34:33.700719 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.700690    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "f3499a34-d2ac-4dbc-ae25-67fde5123af8" (UID: "f3499a34-d2ac-4dbc-ae25-67fde5123af8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:33.702378 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.702354    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t" (OuterVolumeSpecName: "kube-api-access-nlj9t") pod "f3499a34-d2ac-4dbc-ae25-67fde5123af8" (UID: "f3499a34-d2ac-4dbc-ae25-67fde5123af8"). InnerVolumeSpecName "kube-api-access-nlj9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:33.801825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.801793    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f3499a34-d2ac-4dbc-ae25-67fde5123af8-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:33.801825 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.801822    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nlj9t\" (UniqueName: \"kubernetes.io/projected/f3499a34-d2ac-4dbc-ae25-67fde5123af8-kube-api-access-nlj9t\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:33.992766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.992740    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:34:33.994779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:33.994757    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-explicit-292146-5cbbf6ccc8-fjmm9"]
Sep 18 22:34:34.291233 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:34.291158    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2ede803a-1bb9-450b-8074-385fcf65cb82" path="/var/lib/kubelet/pods/2ede803a-1bb9-450b-8074-385fcf65cb82/volumes"
Sep 18 22:34:34.291533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:34.291520    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" path="/var/lib/kubelet/pods/f3499a34-d2ac-4dbc-ae25-67fde5123af8/volumes"
Sep 18 22:34:38.129567 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.129530    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"]
Sep 18 22:34:38.130023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130004    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2ede803a-1bb9-450b-8074-385fcf65cb82" containerName="revoke-keys"
Sep 18 22:34:38.130109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130053    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2ede803a-1bb9-450b-8074-385fcf65cb82" containerName="revoke-keys"
Sep 18 22:34:38.130109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130102    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" containerName="payload-pre-processing"
Sep 18 22:34:38.130172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130111    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" containerName="payload-pre-processing"
Sep 18 22:34:38.130224 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130213    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2ede803a-1bb9-450b-8074-385fcf65cb82" containerName="revoke-keys"
Sep 18 22:34:38.130259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.130233    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f3499a34-d2ac-4dbc-ae25-67fde5123af8" containerName="payload-pre-processing"
Sep 18 22:34:38.135046 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.135012    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:38.138141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.138113    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:34:38.149541 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.149516    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"]
Sep 18 22:34:38.245457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.245430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2\" (UID: \"2afeeab0-104e-46a3-8528-1b39b1149c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:38.346512 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.346489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2\" (UID: \"2afeeab0-104e-46a3-8528-1b39b1149c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:38.347128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.347112    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2\" (UID: \"2afeeab0-104e-46a3-8528-1b39b1149c7a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:38.445382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.445351    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:38.715013 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:38.714985    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"]
Sep 18 22:34:38.716863 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:34:38.716834    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2afeeab0_104e_46a3_8528_1b39b1149c7a.slice/crio-af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf WatchSource:0}: Error finding container af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf: Status 404 returned error can't find the container with id af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf
Sep 18 22:34:39.701380 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:39.701297    2551 generic.go:356] "Generic (PLEG): container finished" podID="2afeeab0-104e-46a3-8528-1b39b1149c7a" containerID="f51ac3c73544c60395a6382c05fed283ae7aedd0e50a135413dcfbff43dc7aa5" exitCode=0
Sep 18 22:34:39.701798 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:39.701372    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2" event={"ID":"2afeeab0-104e-46a3-8528-1b39b1149c7a","Type":"ContainerDied","Data":"f51ac3c73544c60395a6382c05fed283ae7aedd0e50a135413dcfbff43dc7aa5"}
Sep 18 22:34:39.701798 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:39.701419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2" event={"ID":"2afeeab0-104e-46a3-8528-1b39b1149c7a","Type":"ContainerStarted","Data":"af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf"}
Sep 18 22:34:40.767742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:40.767702    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" podUID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" containerName="main" containerID="cri-o://f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778" gracePeriod=45
Sep 18 22:34:40.894232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:40.894208    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:40.971613 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:40.971543    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca\") pod \"2afeeab0-104e-46a3-8528-1b39b1149c7a\" (UID: \"2afeeab0-104e-46a3-8528-1b39b1149c7a\") "
Sep 18 22:34:40.971941 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:40.971919    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2afeeab0-104e-46a3-8528-1b39b1149c7a" (UID: "2afeeab0-104e-46a3-8528-1b39b1149c7a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:41.021819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.021783    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:41.072398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072367    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072405    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kmr94\" (UniqueName: \"kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072423    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072460    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072494    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072521    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home\") pod \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\" (UID: \"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0\") "
Sep 18 22:34:41.072847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072792    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:41.072847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072807    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache" (OuterVolumeSpecName: "model-cache") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:41.072952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072877    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home" (OuterVolumeSpecName: "home") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:41.073009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072953    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2afeeab0-104e-46a3-8528-1b39b1149c7a-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.073009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072974    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.073009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.072991    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.074628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.074609    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm" (OuterVolumeSpecName: "dshm") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:41.074713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.074679    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:34:41.074713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.074684    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94" (OuterVolumeSpecName: "kube-api-access-kmr94") pod "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" (UID: "6f2b38e7-7ae9-4c99-9931-7933ec81f8c0"). InnerVolumeSpecName "kube-api-access-kmr94". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:41.173897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.173868    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.173897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.173891    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.173897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.173900    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.174137 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.173909    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kmr94\" (UniqueName: \"kubernetes.io/projected/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0-kube-api-access-kmr94\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:41.714697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.714664    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"
Sep 18 22:34:41.714906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.714669    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2" event={"ID":"2afeeab0-104e-46a3-8528-1b39b1149c7a","Type":"ContainerDied","Data":"af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf"}
Sep 18 22:34:41.714906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.714782    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="af10615e418f1e061e3abd3135634d8132d43d7fd2f163c7858c9a6040a9ddaf"
Sep 18 22:34:41.716129 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.716104    2551 generic.go:356] "Generic (PLEG): container finished" podID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" containerID="f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778" exitCode=0
Sep 18 22:34:41.716263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.716131    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" event={"ID":"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0","Type":"ContainerDied","Data":"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"}
Sep 18 22:34:41.716263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.716162    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs" event={"ID":"6f2b38e7-7ae9-4c99-9931-7933ec81f8c0","Type":"ContainerDied","Data":"83480d4c051ab3d543f635d3f65fe643cbb8bff4c28bf0cf0fdf102ac0ff3610"}
Sep 18 22:34:41.716263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.716162    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"
Sep 18 22:34:41.716263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.716186    2551 scope.go:117] "RemoveContainer" containerID="f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"
Sep 18 22:34:41.732655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.732617    2551 scope.go:117] "RemoveContainer" containerID="f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"
Sep 18 22:34:41.733392 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:34:41.733363    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778\": container with ID starting with f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778 not found: ID does not exist" containerID="f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"
Sep 18 22:34:41.733860 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.733831    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778"} err="failed to get container status \"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778\": rpc error: code = NotFound desc = could not find container \"f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778\": container with ID starting with f807b30d3d536570c539631ed874674aecf96ab2d6468baaad50757ed77bd778 not found: ID does not exist"
Sep 18 22:34:41.754541 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.754517    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:34:41.757369 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:41.757350    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-explicit-292146/explicit-model-292146-kserve-7c984cc5df-ljtrs"]
Sep 18 22:34:42.291275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:42.291244    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" path="/var/lib/kubelet/pods/6f2b38e7-7ae9-4c99-9931-7933ec81f8c0/volumes"
Sep 18 22:34:44.447608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.447569    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:34:44.448215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448196    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2afeeab0-104e-46a3-8528-1b39b1149c7a" containerName="revoke-keys"
Sep 18 22:34:44.448284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448218    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2afeeab0-104e-46a3-8528-1b39b1149c7a" containerName="revoke-keys"
Sep 18 22:34:44.448284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448233    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" containerName="main"
Sep 18 22:34:44.448284 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448242    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" containerName="main"
Sep 18 22:34:44.448492 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448387    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6f2b38e7-7ae9-4c99-9931-7933ec81f8c0" containerName="main"
Sep 18 22:34:44.448492 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.448405    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2afeeab0-104e-46a3-8528-1b39b1149c7a" containerName="revoke-keys"
Sep 18 22:34:44.452722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.452701    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.457274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.457246    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"distinct-w0-c5fdd3-kserve-self-signed-certs\""
Sep 18 22:34:44.464277 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.464248    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:34:44.509786 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509764    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.509912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509851    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.509912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509882    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.509912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.510110 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509916    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9xkcf\" (UniqueName: \"kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.510110 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.509931    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.610788 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.610758    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.610928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.610792    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9xkcf\" (UniqueName: \"kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.610928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.610812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.610928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.610876    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.611060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.610978    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.611060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.611025    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.611358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.611322    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.611358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.611346    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.611509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.611370    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.613091 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.613066    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.613325 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.613309    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.619166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.619145    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9xkcf\" (UniqueName: \"kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf\") pod \"distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") " pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:44.762988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:44.762910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:45.035823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.035799    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:34:45.038441 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:34:45.038408    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod88a218fc_8465_4a4d_9520_7cbc7e195d99.slice/crio-a208ce585ec4b582b9c728b3f4fb41f12a24b0bd77f2ecb1567f3f3366171e53 WatchSource:0}: Error finding container a208ce585ec4b582b9c728b3f4fb41f12a24b0bd77f2ecb1567f3f3366171e53: Status 404 returned error can't find the container with id a208ce585ec4b582b9c728b3f4fb41f12a24b0bd77f2ecb1567f3f3366171e53
Sep 18 22:34:45.743183 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.743142    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" event={"ID":"88a218fc-8465-4a4d-9520-7cbc7e195d99","Type":"ContainerStarted","Data":"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"}
Sep 18 22:34:45.743587 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.743197    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" event={"ID":"88a218fc-8465-4a4d-9520-7cbc7e195d99","Type":"ContainerStarted","Data":"a208ce585ec4b582b9c728b3f4fb41f12a24b0bd77f2ecb1567f3f3366171e53"}
Sep 18 22:34:45.766418 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.766374    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" podStartSLOduration=1.7663591730000001 podStartE2EDuration="1.766359173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:45.763702606 +0000 UTC m=+1077.945287164" watchObservedRunningTime="2026-09-18 22:34:45.766359173 +0000 UTC m=+1077.947943713"
Sep 18 22:34:45.817091 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.817060    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:34:45.817272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:45.817250    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy" containerID="cri-o://c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7" gracePeriod=30
Sep 18 22:34:46.579290 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.579261    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:34:46.582918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.582894    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.585883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.585861    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-noten-233c27-openshift-default-dockercfg-j4995\""
Sep 18 22:34:46.603797 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.603763    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:34:46.633070 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633023    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633094    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69lgz\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633120    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633146    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633175    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633211    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633298 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633284    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.633477 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.633307    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734498    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734547    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69lgz\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734577    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734769 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734778    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734823    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734887    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734914    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734924    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734942    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.734988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.734943    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.735256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.735236    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.735322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.735280    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.735576 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.735551    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.737227 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.737203    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.737451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.737433    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.743190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.743161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-69lgz\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.743330 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.743311    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token\") pod \"e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") " pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:46.893963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:46.893925    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:47.175051 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.174956    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:34:47.178167 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:34:47.178131    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22cf1c4f_1f46_4ba3_a8a3_f9c301aaf176.slice/crio-1464a17a3f4dafbc2e26dda0edd1112c51925a97a309c9f3e37f03490dd88d06 WatchSource:0}: Error finding container 1464a17a3f4dafbc2e26dda0edd1112c51925a97a309c9f3e37f03490dd88d06: Status 404 returned error can't find the container with id 1464a17a3f4dafbc2e26dda0edd1112c51925a97a309c9f3e37f03490dd88d06
Sep 18 22:34:47.180847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.180800    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:34:47.180935 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.180893    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:34:47.180975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.180944    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:34:47.755991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.755959    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" event={"ID":"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176","Type":"ContainerStarted","Data":"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"}
Sep 18 22:34:47.756372 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.756011    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" event={"ID":"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176","Type":"ContainerStarted","Data":"1464a17a3f4dafbc2e26dda0edd1112c51925a97a309c9f3e37f03490dd88d06"}
Sep 18 22:34:47.776942 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.776892    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" podStartSLOduration=1.776879669 podStartE2EDuration="1.776879669s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:34:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:34:47.776088342 +0000 UTC m=+1079.957672890" watchObservedRunningTime="2026-09-18 22:34:47.776879669 +0000 UTC m=+1079.958464215"
Sep 18 22:34:47.894146 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:47.894119    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:48.899888 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:48.899853    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:49.085002 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.084960    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:49.085243 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.085197    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" podUID="60b5104a-cd47-4a20-b9f0-f4085a675163" containerName="payload-pre-processing" containerID="cri-o://f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0" gracePeriod=30
Sep 18 22:34:49.253638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.253605    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"]
Sep 18 22:34:49.256728 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.256698    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-939c7fd8-4nct2"]
Sep 18 22:34:49.339517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.339487    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:49.463817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.463734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lv2n\" (UniqueName: \"kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n\") pod \"60b5104a-cd47-4a20-b9f0-f4085a675163\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") "
Sep 18 22:34:49.463817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.463773    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume\") pod \"60b5104a-cd47-4a20-b9f0-f4085a675163\" (UID: \"60b5104a-cd47-4a20-b9f0-f4085a675163\") "
Sep 18 22:34:49.464269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.464239    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "60b5104a-cd47-4a20-b9f0-f4085a675163" (UID: "60b5104a-cd47-4a20-b9f0-f4085a675163"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:49.465991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.465955    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n" (OuterVolumeSpecName: "kube-api-access-4lv2n") pod "60b5104a-cd47-4a20-b9f0-f4085a675163" (UID: "60b5104a-cd47-4a20-b9f0-f4085a675163"). InnerVolumeSpecName "kube-api-access-4lv2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:49.565489 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.565464    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lv2n\" (UniqueName: \"kubernetes.io/projected/60b5104a-cd47-4a20-b9f0-f4085a675163-kube-api-access-4lv2n\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:49.565489 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.565486    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60b5104a-cd47-4a20-b9f0-f4085a675163-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:49.769373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769291    2551 generic.go:356] "Generic (PLEG): container finished" podID="60b5104a-cd47-4a20-b9f0-f4085a675163" containerID="f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0" exitCode=0
Sep 18 22:34:49.769373 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769336    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"
Sep 18 22:34:49.769585 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769372    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" event={"ID":"60b5104a-cd47-4a20-b9f0-f4085a675163","Type":"ContainerDied","Data":"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"}
Sep 18 22:34:49.769585 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769427    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z" event={"ID":"60b5104a-cd47-4a20-b9f0-f4085a675163","Type":"ContainerDied","Data":"6295b2558a0315cdb5b1b7127357c684139db9b48eb9442bc5736cb97e20a913"}
Sep 18 22:34:49.769585 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769455    2551 scope.go:117] "RemoveContainer" containerID="f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"
Sep 18 22:34:49.769883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.769867    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:49.770975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.770633    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:34:49.778438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.778422    2551 scope.go:117] "RemoveContainer" containerID="f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"
Sep 18 22:34:49.778691 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:34:49.778673    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0\": container with ID starting with f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0 not found: ID does not exist" containerID="f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"
Sep 18 22:34:49.778735 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.778700    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0"} err="failed to get container status \"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0\": rpc error: code = NotFound desc = could not find container \"f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0\": container with ID starting with f0d18bac8e187a392f9a4c111cc843bbc4a0aafd41612f7f09121bdb7bc797e0 not found: ID does not exist"
Sep 18 22:34:49.805948 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.805923    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:49.809696 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:49.809675    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-939c7fd8-559d689cfd-fpl4z"]
Sep 18 22:34:50.291553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:50.291521    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2afeeab0-104e-46a3-8528-1b39b1149c7a" path="/var/lib/kubelet/pods/2afeeab0-104e-46a3-8528-1b39b1149c7a/volumes"
Sep 18 22:34:50.291897 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:50.291863    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="60b5104a-cd47-4a20-b9f0-f4085a675163" path="/var/lib/kubelet/pods/60b5104a-cd47-4a20-b9f0-f4085a675163/volumes"
Sep 18 22:34:51.074117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.074087    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:34:51.178881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.178849    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.178897    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.178921    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.178960    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179007    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179026    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179139    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179188    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179207    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:51.179602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179258    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-blhhq\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq\") pod \"557cf4ce-39dd-4cc2-8307-f22a31709c38\" (UID: \"557cf4ce-39dd-4cc2-8307-f22a31709c38\") "
Sep 18 22:34:51.179602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179314    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data" (OuterVolumeSpecName: "istio-data") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:51.179602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179376    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:51.179602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179443    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:51.179857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179631    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.179857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179654    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.179857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179669    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.179857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179683    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.179857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.179729    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:34:51.181170 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.181142    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token" (OuterVolumeSpecName: "istio-token") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:51.181319 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.181299    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:34:51.181408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.181387    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:34:51.181408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.181392    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq" (OuterVolumeSpecName: "kube-api-access-blhhq") pod "557cf4ce-39dd-4cc2-8307-f22a31709c38" (UID: "557cf4ce-39dd-4cc2-8307-f22a31709c38"). InnerVolumeSpecName "kube-api-access-blhhq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:34:51.281127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.281093    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/557cf4ce-39dd-4cc2-8307-f22a31709c38-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.281127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.281124    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.281127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.281134    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-blhhq\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-kube-api-access-blhhq\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.281345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.281143    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.281345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.281151    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/557cf4ce-39dd-4cc2-8307-f22a31709c38-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:34:51.782998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.782964    2551 generic.go:356] "Generic (PLEG): container finished" podID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerID="c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7" exitCode=0
Sep 18 22:34:51.783405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.783024    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"
Sep 18 22:34:51.783405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.783059    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerDied","Data":"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"}
Sep 18 22:34:51.783405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.783119    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78" event={"ID":"557cf4ce-39dd-4cc2-8307-f22a31709c38","Type":"ContainerDied","Data":"50d3a8aebb40473a17d7ca3f1c527f601a5f84821c42d21a91617804a117db06"}
Sep 18 22:34:51.783405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.783145    2551 scope.go:117] "RemoveContainer" containerID="c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"
Sep 18 22:34:51.793193 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.793179    2551 scope.go:117] "RemoveContainer" containerID="0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"
Sep 18 22:34:51.801543 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.801526    2551 scope.go:117] "RemoveContainer" containerID="c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"
Sep 18 22:34:51.801830 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:34:51.801808    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7\": container with ID starting with c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7 not found: ID does not exist" containerID="c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"
Sep 18 22:34:51.801874 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.801838    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7"} err="failed to get container status \"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7\": rpc error: code = NotFound desc = could not find container \"c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7\": container with ID starting with c4837a4355b395bdaf804fb34ed70aba2c6cc627054852b32cd11398e5898ce7 not found: ID does not exist"
Sep 18 22:34:51.801874 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.801854    2551 scope.go:117] "RemoveContainer" containerID="0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"
Sep 18 22:34:51.802126 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:34:51.802107    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2\": container with ID starting with 0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2 not found: ID does not exist" containerID="0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"
Sep 18 22:34:51.802177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.802132    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2"} err="failed to get container status \"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2\": rpc error: code = NotFound desc = could not find container \"0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2\": container with ID starting with 0c5ebab796191d59e0ead009243f01f2a07340120a475b76f8254241b0f8fef2 not found: ID does not exist"
Sep 18 22:34:51.812868 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.812838    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:34:51.822316 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:51.822292    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-explicit-292146-openshift-default-5f6bc89c68-t5x78"]
Sep 18 22:34:52.291659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:52.291631    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" path="/var/lib/kubelet/pods/557cf4ce-39dd-4cc2-8307-f22a31709c38/volumes"
Sep 18 22:34:54.763579 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:54.763546    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:54.763960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:54.763626    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:54.776382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:54.776357    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:34:54.814198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:34:54.814170    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:35:08.902859 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:08.902823    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:35:11.279545 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279515    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:35:11.279921 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279909    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:11.279976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279923    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:11.279976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279936    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:11.279976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279942    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:11.279976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279950    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="60b5104a-cd47-4a20-b9f0-f4085a675163" containerName="payload-pre-processing"
Sep 18 22:35:11.279976 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.279956    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="60b5104a-cd47-4a20-b9f0-f4085a675163" containerName="payload-pre-processing"
Sep 18 22:35:11.280141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.280013    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="60b5104a-cd47-4a20-b9f0-f4085a675163" containerName="payload-pre-processing"
Sep 18 22:35:11.280141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.280023    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:11.284685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.284668    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.287384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.287360    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w2-8faaa1-openshift-default-dockercfg-6xg7w\""
Sep 18 22:35:11.298735 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.298709    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:35:11.368978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.368945    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.368989    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369084    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369157    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369186    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369216    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369241    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pjc8t\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.369347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.369267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470063 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470018    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470071    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470127    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470207    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470249 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470245    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470291    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470324    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pjc8t\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470369    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470375    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470526 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470487    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470765 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470703    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.470907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.470883    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.472300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.472281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.472765 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.472745    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.485074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.485054    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.494871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.494852    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pjc8t\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t\") pod \"e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") " pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.595070 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.594984    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:11.871487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.871461    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:35:11.872853 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:35:11.872830    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4fd057de_4441_4639_8445_db3979da3e40.slice/crio-efc38bf5acf9f85d94ce7113bf2e6896e0d114be469264fc9bc3d714168d6e84 WatchSource:0}: Error finding container efc38bf5acf9f85d94ce7113bf2e6896e0d114be469264fc9bc3d714168d6e84: Status 404 returned error can't find the container with id efc38bf5acf9f85d94ce7113bf2e6896e0d114be469264fc9bc3d714168d6e84
Sep 18 22:35:11.875008 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.874977    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:11.875085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.875059    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:11.875124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.875089    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:11.901662 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:11.901625    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerStarted","Data":"efc38bf5acf9f85d94ce7113bf2e6896e0d114be469264fc9bc3d714168d6e84"}
Sep 18 22:35:12.908836 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:12.908798    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerStarted","Data":"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"}
Sep 18 22:35:12.933090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:12.933013    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podStartSLOduration=1.9329822829999999 podStartE2EDuration="1.932982283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:35:12.932177031 +0000 UTC m=+1105.113761577" watchObservedRunningTime="2026-09-18 22:35:12.932982283 +0000 UTC m=+1105.114566832"
Sep 18 22:35:13.596127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:13.596093    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:13.600989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:13.600964    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:13.914567 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:13.914534    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:13.915533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:13.915505    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:35.200504 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:35.200468    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:35:45.507245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.507213    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:35:45.507898 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.507879    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="557cf4ce-39dd-4cc2-8307-f22a31709c38" containerName="istio-proxy"
Sep 18 22:35:45.512270 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.512250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.515832 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.515806    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w2-20531f-openshift-default-dockercfg-79s2v\""
Sep 18 22:35:45.526579 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.526557    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:35:45.588400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588369    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2k844\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588430    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588486    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588524    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588576    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588628    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588669    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588692    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.588738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.588721    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690309 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690275    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690309 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690315    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690405    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2k844\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690466    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690499    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690739 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690531    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690739 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690581    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690739 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690653    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690782    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.690958 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690902    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.691079 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.690951    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.691146 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.691114    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.691343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.691323    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.692754 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.692732    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.693098 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.693077    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.702300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.702275    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2k844\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.720082 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.720060    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token\") pod \"e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") " pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:45.823182 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:45.823087    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:46.125800 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:46.125777    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:35:46.128453 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:35:46.128430    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda4abbdf1_c220_4d58_8dcc_c7318c62130b.slice/crio-3a30815a4c786da6feb8cf7e8aff8ac1909ff2f020327ddf63df6bc4bf77760d WatchSource:0}: Error finding container 3a30815a4c786da6feb8cf7e8aff8ac1909ff2f020327ddf63df6bc4bf77760d: Status 404 returned error can't find the container with id 3a30815a4c786da6feb8cf7e8aff8ac1909ff2f020327ddf63df6bc4bf77760d
Sep 18 22:35:46.130559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:46.130530    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:46.130607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:46.130591    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:46.130651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:46.130619    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:47.099760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:47.099722    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerStarted","Data":"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"}
Sep 18 22:35:47.100177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:47.099789    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerStarted","Data":"3a30815a4c786da6feb8cf7e8aff8ac1909ff2f020327ddf63df6bc4bf77760d"}
Sep 18 22:35:47.124518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:47.124465    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podStartSLOduration=2.124445016 podStartE2EDuration="2.124445016s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:35:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:35:47.123252709 +0000 UTC m=+1139.304837255" watchObservedRunningTime="2026-09-18 22:35:47.124445016 +0000 UTC m=+1139.306029562"
Sep 18 22:35:47.823909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:47.823876    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:47.829324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:47.829303    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:48.106250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:48.106157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:48.107227 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:48.107206    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:35:50.117975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.117944    2551 generic.go:356] "Generic (PLEG): container finished" podID="4fd057de-4441-4639-8445-db3979da3e40" containerID="5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f" exitCode=0
Sep 18 22:35:50.118398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.118025    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerDied","Data":"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"}
Sep 18 22:35:50.118460 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.118443    2551 scope.go:117] "RemoveContainer" containerID="5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"
Sep 18 22:35:50.119666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.119636    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:50.119716 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.119706    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:50.119750 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:50.119736    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:35:51.125203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:51.125165    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerStarted","Data":"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"}
Sep 18 22:35:51.595871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:51.595842    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:51.596049 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:51.596011    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 22:35:51.596100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:51.596067    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 22:35:52.595987 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:52.595950    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 22:35:52.596414 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:52.596008    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 22:35:53.595810 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:53.595771    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 22:35:53.595993 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:53.595824    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 22:35:54.595383 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.595348    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 22:35:54.595748 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.595421    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 22:35:54.729746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.729714    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:35:54.733495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.733472    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.739391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.739358    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-20531f-dockercfg-fr2xn\""
Sep 18 22:35:54.739617 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.739600    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-20531f\""
Sep 18 22:35:54.756782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.756754    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:35:54.883398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.883313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.883398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.883358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kr5h\" (UniqueName: \"kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.984969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.984932    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.985173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.984985    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kr5h\" (UniqueName: \"kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.985688 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.985665    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:54.994119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:54.994092    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kr5h\" (UniqueName: \"kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h\") pod \"payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:55.058156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:55.058132    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:55.335414 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:55.332614    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:35:55.595358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:55.595263    2551 patch_prober.go:28] interesting pod/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused" start-of-body=
Sep 18 22:35:55.595358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:55.595325    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.63:15021/healthz/ready\": dial tcp 10.132.0.63:15021: connect: connection refused"
Sep 18 22:35:56.179787 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.179750    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" event={"ID":"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0","Type":"ContainerStarted","Data":"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"}
Sep 18 22:35:56.179998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.179812    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" event={"ID":"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0","Type":"ContainerStarted","Data":"8b36ba31e162b4a85bb5b30d0d7453ce1d1e48c553a9e5a6e2332cf16572dd1d"}
Sep 18 22:35:56.179998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.179898    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:35:56.199466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.199419    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" podStartSLOduration=2.1994044 podStartE2EDuration="2.1994044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:35:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:35:56.197222697 +0000 UTC m=+1148.378807244" watchObservedRunningTime="2026-09-18 22:35:56.1994044 +0000 UTC m=+1148.380988946"
Sep 18 22:35:56.600236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.600149    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:56.600692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.600671    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:56.601364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:56.601343    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:35:59.179054 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.178996    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"]
Sep 18 22:35:59.182912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.182890    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:35:59.185781 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.185758    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:35:59.192111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.192085    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"]
Sep 18 22:35:59.325827 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.325796    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eac715-p25s9\" (UID: \"675b959e-c32a-4f21-9251-0c9a1b64282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:35:59.426963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.426930    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eac715-p25s9\" (UID: \"675b959e-c32a-4f21-9251-0c9a1b64282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:35:59.427625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.427607    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-eac715-p25s9\" (UID: \"675b959e-c32a-4f21-9251-0c9a1b64282b\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:35:59.494967 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.494902    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:35:59.771652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:35:59.771610    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"]
Sep 18 22:35:59.773848 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:35:59.773824    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod675b959e_c32a_4f21_9251_0c9a1b64282b.slice/crio-babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885 WatchSource:0}: Error finding container babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885: Status 404 returned error can't find the container with id babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885
Sep 18 22:36:00.204618 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:00.204585    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9" event={"ID":"675b959e-c32a-4f21-9251-0c9a1b64282b","Type":"ContainerStarted","Data":"41eb53e0de0caaa8446c573730f3fc8f3654611792555a19c60c8ac3babe7b59"}
Sep 18 22:36:00.204924 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:00.204640    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9" event={"ID":"675b959e-c32a-4f21-9251-0c9a1b64282b","Type":"ContainerStarted","Data":"babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885"}
Sep 18 22:36:00.221349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:00.221301    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9" podStartSLOduration=1.221285641 podStartE2EDuration="1.221285641s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:35:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:36:00.220640312 +0000 UTC m=+1152.402224858" watchObservedRunningTime="2026-09-18 22:36:00.221285641 +0000 UTC m=+1152.402870187"
Sep 18 22:36:01.211228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:01.211194    2551 generic.go:356] "Generic (PLEG): container finished" podID="675b959e-c32a-4f21-9251-0c9a1b64282b" containerID="41eb53e0de0caaa8446c573730f3fc8f3654611792555a19c60c8ac3babe7b59" exitCode=0
Sep 18 22:36:01.211625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:01.211294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9" event={"ID":"675b959e-c32a-4f21-9251-0c9a1b64282b","Type":"ContainerDied","Data":"41eb53e0de0caaa8446c573730f3fc8f3654611792555a19c60c8ac3babe7b59"}
Sep 18 22:36:02.358724 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:02.358687    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:36:02.458670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:02.458639    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca\") pod \"675b959e-c32a-4f21-9251-0c9a1b64282b\" (UID: \"675b959e-c32a-4f21-9251-0c9a1b64282b\") "
Sep 18 22:36:02.458990 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:02.458969    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "675b959e-c32a-4f21-9251-0c9a1b64282b" (UID: "675b959e-c32a-4f21-9251-0c9a1b64282b"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:02.559973 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:02.559894    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/675b959e-c32a-4f21-9251-0c9a1b64282b-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:03.227532 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:03.227497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9" event={"ID":"675b959e-c32a-4f21-9251-0c9a1b64282b","Type":"ContainerDied","Data":"babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885"}
Sep 18 22:36:03.227695 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:03.227575    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="babda4a0987f4f25f31bcb68481bc6192b4f1a6fad01ba988f7452a29e65a885"
Sep 18 22:36:03.227695 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:03.227509    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"
Sep 18 22:36:07.187466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:07.187435    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:36:09.626250 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.626209    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7541-324da0"]
Sep 18 22:36:09.626833 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.626810    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="675b959e-c32a-4f21-9251-0c9a1b64282b" containerName="revoke-keys"
Sep 18 22:36:09.626924 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.626835    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="675b959e-c32a-4f21-9251-0c9a1b64282b" containerName="revoke-keys"
Sep 18 22:36:09.626988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.626934    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="675b959e-c32a-4f21-9251-0c9a1b64282b" containerName="revoke-keys"
Sep 18 22:36:09.630204 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.630183    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:09.632823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.632798    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lgx4p\""
Sep 18 22:36:09.635389 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.635362    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-324da0"]
Sep 18 22:36:09.732857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.732828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97nmp\" (UniqueName: \"kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp\") pod \"test-curl-7541-324da0\" (UID: \"876822f7-21b6-4aa2-a80a-0fa7f26325fd\") " pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:09.834232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.834195    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97nmp\" (UniqueName: \"kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp\") pod \"test-curl-7541-324da0\" (UID: \"876822f7-21b6-4aa2-a80a-0fa7f26325fd\") " pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:09.842538 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.842507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97nmp\" (UniqueName: \"kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp\") pod \"test-curl-7541-324da0\" (UID: \"876822f7-21b6-4aa2-a80a-0fa7f26325fd\") " pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:09.942483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:09.942462    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:10.214608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:10.214584    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7541-324da0"]
Sep 18 22:36:10.216180 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:36:10.216151    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod876822f7_21b6_4aa2_a80a_0fa7f26325fd.slice/crio-3a5a4913d9db910635e08d986f14dfe97e3a58d492ac833acbff1f942c9f1014 WatchSource:0}: Error finding container 3a5a4913d9db910635e08d986f14dfe97e3a58d492ac833acbff1f942c9f1014: Status 404 returned error can't find the container with id 3a5a4913d9db910635e08d986f14dfe97e3a58d492ac833acbff1f942c9f1014
Sep 18 22:36:10.270063 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:10.270024    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-324da0" event={"ID":"876822f7-21b6-4aa2-a80a-0fa7f26325fd","Type":"ContainerStarted","Data":"3a5a4913d9db910635e08d986f14dfe97e3a58d492ac833acbff1f942c9f1014"}
Sep 18 22:36:11.209926 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.209845    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"]
Sep 18 22:36:11.213277 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.213252    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-eac715-p25s9"]
Sep 18 22:36:11.276963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.276927    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7541-324da0" event={"ID":"876822f7-21b6-4aa2-a80a-0fa7f26325fd","Type":"ContainerStarted","Data":"81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c"}
Sep 18 22:36:11.295008 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.294950    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7541-324da0" podStartSLOduration=2.294930292 podStartE2EDuration="2.294930292s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:36:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:36:11.293505308 +0000 UTC m=+1163.475089856" watchObservedRunningTime="2026-09-18 22:36:11.294930292 +0000 UTC m=+1163.476514839"
Sep 18 22:36:11.892557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.892515    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7541-324da0"]
Sep 18 22:36:11.897023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:11.896996    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7541-324da0"]
Sep 18 22:36:12.282851 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:36:12.282815    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7541-324da0" container="test-curl-7541-324da0"
Sep 18 22:36:12.290859 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:12.290833    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="675b959e-c32a-4f21-9251-0c9a1b64282b" path="/var/lib/kubelet/pods/675b959e-c32a-4f21-9251-0c9a1b64282b/volumes"
Sep 18 22:36:13.288194 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:13.288155    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7541-324da0" podUID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" containerName="test-curl-7541-324da0" containerID="cri-o://81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c" gracePeriod=2
Sep 18 22:36:15.390486 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:15.390448    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:36:15.467995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:15.467963    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:15.488194 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:15.488171    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-97nmp\" (UniqueName: \"kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp\") pod \"876822f7-21b6-4aa2-a80a-0fa7f26325fd\" (UID: \"876822f7-21b6-4aa2-a80a-0fa7f26325fd\") "
Sep 18 22:36:15.490266 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:15.490242    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp" (OuterVolumeSpecName: "kube-api-access-97nmp") pod "876822f7-21b6-4aa2-a80a-0fa7f26325fd" (UID: "876822f7-21b6-4aa2-a80a-0fa7f26325fd"). InnerVolumeSpecName "kube-api-access-97nmp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:15.589918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:15.589841    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-97nmp\" (UniqueName: \"kubernetes.io/projected/876822f7-21b6-4aa2-a80a-0fa7f26325fd-kube-api-access-97nmp\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:16.293142 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.293101    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" path="/var/lib/kubelet/pods/876822f7-21b6-4aa2-a80a-0fa7f26325fd/volumes"
Sep 18 22:36:16.310013 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.309985    2551 generic.go:356] "Generic (PLEG): container finished" podID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" containerID="81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c" exitCode=137
Sep 18 22:36:16.310185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.310172    2551 scope.go:117] "RemoveContainer" containerID="81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c"
Sep 18 22:36:16.310308 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.310286    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7541-324da0"
Sep 18 22:36:16.319176 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.319155    2551 scope.go:117] "RemoveContainer" containerID="81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c"
Sep 18 22:36:16.319448 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:36:16.319429    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c\": container with ID starting with 81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c not found: ID does not exist" containerID="81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c"
Sep 18 22:36:16.319503 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:16.319457    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c"} err="failed to get container status \"81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c\": rpc error: code = NotFound desc = could not find container \"81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c\": container with ID starting with 81f9db2d32b1e70147649a9cfae7825b3e8c721a50c2f0692c740618a68c207c not found: ID does not exist"
Sep 18 22:36:25.044198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.044167    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"]
Sep 18 22:36:25.044629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.044614    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" containerName="test-curl-7541-324da0"
Sep 18 22:36:25.044629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.044629    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" containerName="test-curl-7541-324da0"
Sep 18 22:36:25.044705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.044698    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="876822f7-21b6-4aa2-a80a-0fa7f26325fd" containerName="test-curl-7541-324da0"
Sep 18 22:36:25.047993 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.047978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:25.050408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.050385    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:36:25.057092 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.057068    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"]
Sep 18 22:36:25.191932 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.191902    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44d735-92ftz\" (UID: \"75c22068-3521-4efe-a4a3-630b841fd4e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:25.293367 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.293334    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44d735-92ftz\" (UID: \"75c22068-3521-4efe-a4a3-630b841fd4e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:25.293909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.293890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-44d735-92ftz\" (UID: \"75c22068-3521-4efe-a4a3-630b841fd4e5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:25.361199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.361140    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:25.648449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:25.648414    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"]
Sep 18 22:36:25.650777 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:36:25.650751    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod75c22068_3521_4efe_a4a3_630b841fd4e5.slice/crio-5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1 WatchSource:0}: Error finding container 5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1: Status 404 returned error can't find the container with id 5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1
Sep 18 22:36:26.382093 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:26.382068    2551 generic.go:356] "Generic (PLEG): container finished" podID="75c22068-3521-4efe-a4a3-630b841fd4e5" containerID="6566968b01518f732ee3795ebd0e85fc660a7c0fb6cfdcf5b959a18841dbadf5" exitCode=0
Sep 18 22:36:26.382452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:26.382129    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz" event={"ID":"75c22068-3521-4efe-a4a3-630b841fd4e5","Type":"ContainerDied","Data":"6566968b01518f732ee3795ebd0e85fc660a7c0fb6cfdcf5b959a18841dbadf5"}
Sep 18 22:36:26.382452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:26.382167    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz" event={"ID":"75c22068-3521-4efe-a4a3-630b841fd4e5","Type":"ContainerStarted","Data":"5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1"}
Sep 18 22:36:27.390906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.390823    2551 generic.go:356] "Generic (PLEG): container finished" podID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerID="fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe" exitCode=0
Sep 18 22:36:27.390906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.390895    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerDied","Data":"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"}
Sep 18 22:36:27.391841 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.391519    2551 scope.go:117] "RemoveContainer" containerID="fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"
Sep 18 22:36:27.392899 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.392864    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:36:27.392980 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.392947    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:36:27.393018 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.392981    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:36:27.539753 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.539722    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:27.617397 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.617367    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca\") pod \"75c22068-3521-4efe-a4a3-630b841fd4e5\" (UID: \"75c22068-3521-4efe-a4a3-630b841fd4e5\") "
Sep 18 22:36:27.617685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.617664    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "75c22068-3521-4efe-a4a3-630b841fd4e5" (UID: "75c22068-3521-4efe-a4a3-630b841fd4e5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:27.617744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:27.617733    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/75c22068-3521-4efe-a4a3-630b841fd4e5-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:28.398436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.398400    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerStarted","Data":"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"}
Sep 18 22:36:28.399819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.399800    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"
Sep 18 22:36:28.399819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.399808    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz" event={"ID":"75c22068-3521-4efe-a4a3-630b841fd4e5","Type":"ContainerDied","Data":"5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1"}
Sep 18 22:36:28.399961 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.399846    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5e0453f7fbe8745ba45e39af87c3598227b3b0968b2373de049f5f8f3f314df1"
Sep 18 22:36:28.823651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.823617    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:36:28.824812 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.824788    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:28.824869 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:28.824840    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:29.823912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:29.823875    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:29.824291 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:29.823946    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:30.824021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:30.823987    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:30.824415 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:30.824086    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:31.102498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:31.102415    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:36:31.102655 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:31.102628    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerName="istio-proxy" containerID="cri-o://86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd" gracePeriod=30
Sep 18 22:36:31.824369 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:31.824332    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:31.824786 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:31.824389    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:32.824154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:32.824118    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:32.824331 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:32.824170    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:33.107191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:33.107096    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:36:33.823568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:33.823530    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused" start-of-body=
Sep 18 22:36:33.823862 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:33.823595    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.64:15021/healthz/ready\": dial tcp 10.132.0.64:15021: connect: connection refused"
Sep 18 22:36:34.772415 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:34.772373    2551 patch_prober.go:28] interesting pod/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:36:34.772831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:34.772440    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:36:34.827179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:34.827155    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:36:34.827928 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:34.827910    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:36:35.876072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:35.876024    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:36:35.876640 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:35.876610    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" podUID="6dc073aa-87cf-4a31-ace1-25156fabd404" containerName="payload-processing" containerID="cri-o://561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea" gracePeriod=30
Sep 18 22:36:35.881201 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:35.881176    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:36:35.881466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:35.881439    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" podUID="baafe798-42d1-4556-9636-25647bafd06b" containerName="payload-pre-processing" containerID="cri-o://45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d" gracePeriod=30
Sep 18 22:36:36.068274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.068245    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"]
Sep 18 22:36:36.072340 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.072317    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-44d735-92ftz"]
Sep 18 22:36:36.227454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.227426    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:36:36.232884 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.232861    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:36:36.292799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.292767    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="75c22068-3521-4efe-a4a3-630b841fd4e5" path="/var/lib/kubelet/pods/75c22068-3521-4efe-a4a3-630b841fd4e5/volumes"
Sep 18 22:36:36.379350 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.379322    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:36:36.408995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.408969    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume\") pod \"6dc073aa-87cf-4a31-ace1-25156fabd404\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") "
Sep 18 22:36:36.409119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409004    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume\") pod \"baafe798-42d1-4556-9636-25647bafd06b\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") "
Sep 18 22:36:36.409119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409059    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jc97h\" (UniqueName: \"kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h\") pod \"6dc073aa-87cf-4a31-ace1-25156fabd404\" (UID: \"6dc073aa-87cf-4a31-ace1-25156fabd404\") "
Sep 18 22:36:36.409251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409191    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-nrqvh\" (UniqueName: \"kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh\") pod \"baafe798-42d1-4556-9636-25647bafd06b\" (UID: \"baafe798-42d1-4556-9636-25647bafd06b\") "
Sep 18 22:36:36.409408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409379    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "baafe798-42d1-4556-9636-25647bafd06b" (UID: "baafe798-42d1-4556-9636-25647bafd06b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:36.409408 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409397    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6dc073aa-87cf-4a31-ace1-25156fabd404" (UID: "6dc073aa-87cf-4a31-ace1-25156fabd404"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:36.409595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409579    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6dc073aa-87cf-4a31-ace1-25156fabd404-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.409661 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.409596    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/baafe798-42d1-4556-9636-25647bafd06b-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.411131 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.411105    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h" (OuterVolumeSpecName: "kube-api-access-jc97h") pod "6dc073aa-87cf-4a31-ace1-25156fabd404" (UID: "6dc073aa-87cf-4a31-ace1-25156fabd404"). InnerVolumeSpecName "kube-api-access-jc97h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:36.411588 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.411570    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh" (OuterVolumeSpecName: "kube-api-access-nrqvh") pod "baafe798-42d1-4556-9636-25647bafd06b" (UID: "baafe798-42d1-4556-9636-25647bafd06b"). InnerVolumeSpecName "kube-api-access-nrqvh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:36.450076 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.449982    2551 generic.go:356] "Generic (PLEG): container finished" podID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerID="86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd" exitCode=0
Sep 18 22:36:36.450076 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.450048    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"
Sep 18 22:36:36.450286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.450074    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" event={"ID":"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176","Type":"ContainerDied","Data":"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"}
Sep 18 22:36:36.450286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.450132    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc" event={"ID":"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176","Type":"ContainerDied","Data":"1464a17a3f4dafbc2e26dda0edd1112c51925a97a309c9f3e37f03490dd88d06"}
Sep 18 22:36:36.450286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.450164    2551 scope.go:117] "RemoveContainer" containerID="86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"
Sep 18 22:36:36.451618 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.451596    2551 generic.go:356] "Generic (PLEG): container finished" podID="baafe798-42d1-4556-9636-25647bafd06b" containerID="45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d" exitCode=0
Sep 18 22:36:36.451733 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.451621    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" event={"ID":"baafe798-42d1-4556-9636-25647bafd06b","Type":"ContainerDied","Data":"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"}
Sep 18 22:36:36.451733 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.451653    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"
Sep 18 22:36:36.451733 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.451663    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w" event={"ID":"baafe798-42d1-4556-9636-25647bafd06b","Type":"ContainerDied","Data":"6a5a26bd81ad8764224dc10530cb93db10e890156356cd0f33b2e1c048cf2daa"}
Sep 18 22:36:36.453114 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.453094    2551 generic.go:356] "Generic (PLEG): container finished" podID="6dc073aa-87cf-4a31-ace1-25156fabd404" containerID="561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea" exitCode=0
Sep 18 22:36:36.453245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.453133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" event={"ID":"6dc073aa-87cf-4a31-ace1-25156fabd404","Type":"ContainerDied","Data":"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"}
Sep 18 22:36:36.453245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.453141    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"
Sep 18 22:36:36.453245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.453166    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn" event={"ID":"6dc073aa-87cf-4a31-ace1-25156fabd404","Type":"ContainerDied","Data":"c7dedbb4b5ff35b12f904e9a9b976d3f76b28bc289ae08b1dc0336a606d452bc"}
Sep 18 22:36:36.460287 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.460267    2551 scope.go:117] "RemoveContainer" containerID="86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"
Sep 18 22:36:36.460613 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:36:36.460582    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd\": container with ID starting with 86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd not found: ID does not exist" containerID="86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"
Sep 18 22:36:36.460685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.460621    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd"} err="failed to get container status \"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd\": rpc error: code = NotFound desc = could not find container \"86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd\": container with ID starting with 86ed46f207a41e8a0d52e2a709a727afa7276ea19f02535a6263f819c4d37ffd not found: ID does not exist"
Sep 18 22:36:36.460685 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.460645    2551 scope.go:117] "RemoveContainer" containerID="45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"
Sep 18 22:36:36.469713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.469697    2551 scope.go:117] "RemoveContainer" containerID="45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"
Sep 18 22:36:36.469956 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:36:36.469937    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d\": container with ID starting with 45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d not found: ID does not exist" containerID="45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"
Sep 18 22:36:36.469998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.469964    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d"} err="failed to get container status \"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d\": rpc error: code = NotFound desc = could not find container \"45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d\": container with ID starting with 45e23928158603b5f478bd87626d27890bc66b8334f419ca4dfc7789efd6511d not found: ID does not exist"
Sep 18 22:36:36.469998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.469980    2551 scope.go:117] "RemoveContainer" containerID="561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"
Sep 18 22:36:36.478180 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.478133    2551 scope.go:117] "RemoveContainer" containerID="561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"
Sep 18 22:36:36.478410 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:36:36.478391    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea\": container with ID starting with 561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea not found: ID does not exist" containerID="561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"
Sep 18 22:36:36.478454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.478418    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea"} err="failed to get container status \"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea\": rpc error: code = NotFound desc = could not find container \"561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea\": container with ID starting with 561af93f3a10a9f89b579aaccafc772358b41262cc99281c7137beb6a19753ea not found: ID does not exist"
Sep 18 22:36:36.487282 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.487261    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:36:36.491894 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.491874    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-44d735-7b59bb74fd-fk22w"]
Sep 18 22:36:36.509528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.509502    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:36:36.510338 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510320    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510424 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510363    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510424 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510409    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510535 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510466    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510591 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510536    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510591 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510563    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510616    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510651    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-69lgz\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510673    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:36:36.510743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket\") pod \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\" (UID: \"22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176\") "
Sep 18 22:36:36.510929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510675    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:36.510929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510762    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data" (OuterVolumeSpecName: "istio-data") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:36.510929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510808    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:36.511089 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.510986    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:36.511089 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511080    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.511198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511101    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nrqvh\" (UniqueName: \"kubernetes.io/projected/baafe798-42d1-4556-9636-25647bafd06b-kube-api-access-nrqvh\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.511198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511116    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.511198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511126    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.511198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511134    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jc97h\" (UniqueName: \"kubernetes.io/projected/6dc073aa-87cf-4a31-ace1-25156fabd404-kube-api-access-jc97h\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.511198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.511143    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.512443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.512420    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:36:36.512804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.512784    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz" (OuterVolumeSpecName: "kube-api-access-69lgz") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "kube-api-access-69lgz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:36.513066 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.513047    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:36:36.513145 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.513066    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token" (OuterVolumeSpecName: "istio-token") pod "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" (UID: "22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:36:36.516945 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.516926    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-44d735-5b9bdd8c89-2xkrn"]
Sep 18 22:36:36.612177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.612155    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.612177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.612175    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.612326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.612184    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.612326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.612192    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69lgz\" (UniqueName: \"kubernetes.io/projected/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-kube-api-access-69lgz\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.612326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.612201    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:36:36.777166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.777140    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:36:36.785968 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:36.785946    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-noten-233c27-openshift-default-685cb96dcb-vzsgc"]
Sep 18 22:36:38.293543 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:38.293511    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" path="/var/lib/kubelet/pods/22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176/volumes"
Sep 18 22:36:38.293902 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:38.293879    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6dc073aa-87cf-4a31-ace1-25156fabd404" path="/var/lib/kubelet/pods/6dc073aa-87cf-4a31-ace1-25156fabd404/volumes"
Sep 18 22:36:38.294215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:38.294201    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="baafe798-42d1-4556-9636-25647bafd06b" path="/var/lib/kubelet/pods/baafe798-42d1-4556-9636-25647bafd06b/volumes"
Sep 18 22:36:48.317966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:48.317942    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:36:48.318443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:48.317942    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:36:48.416117 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:48.416087    2551 scope.go:117] "RemoveContainer" containerID="ed98a95554f207b32b894c064cdb66c43e3e24b6bd09ac6920aa8f1df7fca138"
Sep 18 22:36:48.424313 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:48.424293    2551 scope.go:117] "RemoveContainer" containerID="49bfc854e51875cc537742f85938532991d6029f19275d0b55ddf1e6cf6969ac"
Sep 18 22:36:49.312719 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:49.312687    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:36:52.714941 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.714906    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:36:52.715368 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715352    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6dc073aa-87cf-4a31-ace1-25156fabd404" containerName="payload-processing"
Sep 18 22:36:52.715420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715371    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6dc073aa-87cf-4a31-ace1-25156fabd404" containerName="payload-processing"
Sep 18 22:36:52.715420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715391    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerName="istio-proxy"
Sep 18 22:36:52.715420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715397    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerName="istio-proxy"
Sep 18 22:36:52.715420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715414    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="baafe798-42d1-4556-9636-25647bafd06b" containerName="payload-pre-processing"
Sep 18 22:36:52.715420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715419    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="baafe798-42d1-4556-9636-25647bafd06b" containerName="payload-pre-processing"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715426    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="75c22068-3521-4efe-a4a3-630b841fd4e5" containerName="revoke-keys"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715432    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="75c22068-3521-4efe-a4a3-630b841fd4e5" containerName="revoke-keys"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715492    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="75c22068-3521-4efe-a4a3-630b841fd4e5" containerName="revoke-keys"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715501    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="22cf1c4f-1f46-4ba3-a8a3-f9c301aaf176" containerName="istio-proxy"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715507    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="baafe798-42d1-4556-9636-25647bafd06b" containerName="payload-pre-processing"
Sep 18 22:36:52.715599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.715512    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6dc073aa-87cf-4a31-ace1-25156fabd404" containerName="payload-processing"
Sep 18 22:36:52.720405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.720387    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.723413 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.723372    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-d6c0ac\""
Sep 18 22:36:52.749181 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.749155    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:36:52.752942 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.752920    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:36:52.753065 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.753025    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.756054 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.756016    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-d6c0ac-dockercfg-rsmz2\""
Sep 18 22:36:52.767169 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.767147    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vcmf\" (UniqueName: \"kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.767254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.767221    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.771586 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.771563    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:36:52.868780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.868745    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.868935 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.868796    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xj6l\" (UniqueName: \"kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.868935 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.868878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vcmf\" (UniqueName: \"kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.869007 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.868962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.869703 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.869685    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.881409 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.881384    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vcmf\" (UniqueName: \"kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf\") pod \"payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:52.969878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.969796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.969878 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.969875    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xj6l\" (UniqueName: \"kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.970610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.970589    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:52.979394 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:52.979370    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xj6l\" (UniqueName: \"kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l\") pod \"payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:53.031304 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.031277    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:53.063580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.063555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:53.308889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.308865    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:36:53.310589 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:36:53.310563    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod742d9377_2264_4c69_92d3_b005cf092233.slice/crio-b1970465e6280fa7b296b5ed81b47ae00302eede818b77d5d86d6ba35a4e92ff WatchSource:0}: Error finding container b1970465e6280fa7b296b5ed81b47ae00302eede818b77d5d86d6ba35a4e92ff: Status 404 returned error can't find the container with id b1970465e6280fa7b296b5ed81b47ae00302eede818b77d5d86d6ba35a4e92ff
Sep 18 22:36:53.342166 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.342143    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:36:53.344130 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:36:53.344097    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee186927_348d_496a_bfe8_9b67a48d5ee4.slice/crio-8054179b237896fefab912447aa6b7f69af1c8467848215f84a04d2b891712ac WatchSource:0}: Error finding container 8054179b237896fefab912447aa6b7f69af1c8467848215f84a04d2b891712ac: Status 404 returned error can't find the container with id 8054179b237896fefab912447aa6b7f69af1c8467848215f84a04d2b891712ac
Sep 18 22:36:53.556428 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.556395    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" event={"ID":"ee186927-348d-496a-bfe8-9b67a48d5ee4","Type":"ContainerStarted","Data":"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"}
Sep 18 22:36:53.556595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.556458    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:36:53.556595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.556485    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" event={"ID":"ee186927-348d-496a-bfe8-9b67a48d5ee4","Type":"ContainerStarted","Data":"8054179b237896fefab912447aa6b7f69af1c8467848215f84a04d2b891712ac"}
Sep 18 22:36:53.561082 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.560978    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" event={"ID":"742d9377-2264-4c69-92d3-b005cf092233","Type":"ContainerStarted","Data":"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"}
Sep 18 22:36:53.561294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.561266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" event={"ID":"742d9377-2264-4c69-92d3-b005cf092233","Type":"ContainerStarted","Data":"b1970465e6280fa7b296b5ed81b47ae00302eede818b77d5d86d6ba35a4e92ff"}
Sep 18 22:36:53.561845 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.561810    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:36:53.602737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.602600    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" podStartSLOduration=1.602588083 podStartE2EDuration="1.602588083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:36:53.60128872 +0000 UTC m=+1205.782873262" watchObservedRunningTime="2026-09-18 22:36:53.602588083 +0000 UTC m=+1205.784172642"
Sep 18 22:36:53.648092 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:36:53.648021    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" podStartSLOduration=1.648006227 podStartE2EDuration="1.648006227s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:36:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:36:53.646836413 +0000 UTC m=+1205.828420970" watchObservedRunningTime="2026-09-18 22:36:53.648006227 +0000 UTC m=+1205.829590773"
Sep 18 22:37:04.568621 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:04.568589    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:37:04.569077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:04.568760    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:37:17.530667 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.530634    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:37:17.540516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.540483    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.544584 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.544561    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w3-a91c21-openshift-default-dockercfg-8j8sm\""
Sep 18 22:37:17.559371 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.559345    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:37:17.694376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694346    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694421    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694494    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dzdwq\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694528    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694555    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694624    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694723    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.694830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.694783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796366    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796413    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796466    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796561    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dzdwq\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796590    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796686    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796775    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796855    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.796989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.796908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.797171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.797076    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.797469 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.797450    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.798814 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.798785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.798950 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.798933    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.805177 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.805150    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.805487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.805465    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dzdwq\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq\") pod \"e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") " pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:17.852449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:17.852424    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:18.139806 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.139777    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:37:18.141531 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:37:18.141502    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13d7000e_aef2_474d_bbee_cce25facbae7.slice/crio-24455a421e656064c4bb0777e0caeac8b340ee153d773211c40a879ec233b370 WatchSource:0}: Error finding container 24455a421e656064c4bb0777e0caeac8b340ee153d773211c40a879ec233b370: Status 404 returned error can't find the container with id 24455a421e656064c4bb0777e0caeac8b340ee153d773211c40a879ec233b370
Sep 18 22:37:18.147493 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.147455    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:18.147610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.147522    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:18.147610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.147548    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:18.706387 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.706353    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerStarted","Data":"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"}
Sep 18 22:37:18.706884 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.706413    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerStarted","Data":"24455a421e656064c4bb0777e0caeac8b340ee153d773211c40a879ec233b370"}
Sep 18 22:37:18.730952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.730898    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podStartSLOduration=1.730880582 podStartE2EDuration="1.730880582s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:37:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:37:18.730520422 +0000 UTC m=+1230.912104970" watchObservedRunningTime="2026-09-18 22:37:18.730880582 +0000 UTC m=+1230.912465130"
Sep 18 22:37:18.852780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:18.852744    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:19.857628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:19.857600    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:20.717497 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:20.717466    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:20.718374 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:20.718349    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:27.398108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:27.392595    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:37:48.465107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:48.465013    2551 scope.go:117] "RemoveContainer" containerID="6382b2c87bee100bebd01015e1dd265d3eb3d219f173d498ffb75e06a250baea"
Sep 18 22:37:54.918920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.918883    2551 generic.go:356] "Generic (PLEG): container finished" podID="13d7000e-aef2-474d-bbee-cce25facbae7" containerID="b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d" exitCode=0
Sep 18 22:37:54.919473 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.918965    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerDied","Data":"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"}
Sep 18 22:37:54.919549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.919500    2551 scope.go:117] "RemoveContainer" containerID="b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"
Sep 18 22:37:54.920769 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.920738    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:54.920869 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.920808    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:54.920869 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:54.920839    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:37:55.926023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:55.925981    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerStarted","Data":"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"}
Sep 18 22:37:56.853487 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:56.853381    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:37:56.854207 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:56.854179    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:37:56.854403 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:56.854375    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:37:57.853306 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:57.853269    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:37:57.853900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:57.853866    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:37:58.853746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:58.853698    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:37:58.854265 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:58.853767    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:37:59.109981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:59.109869    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:37:59.853480 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:59.853445    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:37:59.853798 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:37:59.853758    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:38:00.853663 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:00.853179    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:38:00.853663 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:00.853257    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:38:01.853664 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:01.853619    2551 patch_prober.go:28] interesting pod/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused" start-of-body=
Sep 18 22:38:01.853855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:01.853689    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.71:15021/healthz/ready\": dial tcp 10.132.0.71:15021: connect: connection refused"
Sep 18 22:38:02.857540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:02.857513    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:38:02.857906 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:02.857787    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:38:02.858691 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:02.858663    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:38:08.186995 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.186963    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:38:08.189502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.189483    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.193787 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.193757    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-20e1128a\"/\"e2e-lifecycle-model-20e1128a-kserve-self-signed-certs\""
Sep 18 22:38:08.193912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.193789    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-20e1128a\"/\"kube-root-ca.crt\""
Sep 18 22:38:08.193912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.193830    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-mt-20e1128a\"/\"default-dockercfg-jhzjd\""
Sep 18 22:38:08.193912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.193769    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-mt-20e1128a\"/\"openshift-service-ca.crt\""
Sep 18 22:38:08.214660 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.214623    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:38:08.273624 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273591    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.273780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273638    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.273780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273690    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.273780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273729    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.273917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273808    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4v2k\" (UniqueName: \"kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.273917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.273894    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375151 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375118    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4v2k\" (UniqueName: \"kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375335 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375225    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375335 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375293    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375335 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375332    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375372    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375416    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.375811 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375787    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.376139 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.375984    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.376139 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.376055    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.380056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.380008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.380605 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.380323    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.385568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.385536    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4v2k\" (UniqueName: \"kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k\") pod \"e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") " pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.498718 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.498645    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:08.781000 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:08.780973    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:38:08.783072 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:38:08.783016    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-poddf6e8028_6c60_4aba_a9c8_97b0647e8ec5.slice/crio-d03d500e032d3a305fa31b63d0a31369d7af6c846db71b20f1edaac1b0a0cd53 WatchSource:0}: Error finding container d03d500e032d3a305fa31b63d0a31369d7af6c846db71b20f1edaac1b0a0cd53: Status 404 returned error can't find the container with id d03d500e032d3a305fa31b63d0a31369d7af6c846db71b20f1edaac1b0a0cd53
Sep 18 22:38:09.014214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:09.014180    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" event={"ID":"df6e8028-6c60-4aba-a9c8-97b0647e8ec5","Type":"ContainerStarted","Data":"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"}
Sep 18 22:38:09.014402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:09.014223    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" event={"ID":"df6e8028-6c60-4aba-a9c8-97b0647e8ec5","Type":"ContainerStarted","Data":"d03d500e032d3a305fa31b63d0a31369d7af6c846db71b20f1edaac1b0a0cd53"}
Sep 18 22:38:18.499563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:18.499530    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:18.500017 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:18.499578    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:18.511791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:18.511767    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:18.532144 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:18.532101    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" podStartSLOduration=10.532086511 podStartE2EDuration="10.532086511s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:38:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:38:09.049479254 +0000 UTC m=+1281.231063800" watchObservedRunningTime="2026-09-18 22:38:18.532086511 +0000 UTC m=+1290.713671110"
Sep 18 22:38:19.083715 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:19.083682    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:38:22.919141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:22.919108    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:38:56.199597 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:38:56.199562    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:39:00.779638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.779603    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"]
Sep 18 22:39:00.782255 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.782236    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:00.785742 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.785719    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:39:00.795780 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.795759    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"]
Sep 18 22:39:00.863978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.863950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf\" (UID: \"f73e34b0-40c7-41b2-82fc-ea3837f439c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:00.965225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.965189    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf\" (UID: \"f73e34b0-40c7-41b2-82fc-ea3837f439c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:00.965810 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:00.965791    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf\" (UID: \"f73e34b0-40c7-41b2-82fc-ea3837f439c1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:01.092686 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:01.092582    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:01.376110 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:01.376084    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"]
Sep 18 22:39:01.378742 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:39:01.378713    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73e34b0_40c7_41b2_82fc_ea3837f439c1.slice/crio-c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d WatchSource:0}: Error finding container c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d: Status 404 returned error can't find the container with id c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d
Sep 18 22:39:02.149615 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:39:02.149590    2551 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf73e34b0_40c7_41b2_82fc_ea3837f439c1.slice/crio-conmon-330b45b0374f544e49a26ed2438d0cdf5a605fc112db24808f22379d1a0de472.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 22:39:02.313888 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:02.313853    2551 generic.go:356] "Generic (PLEG): container finished" podID="f73e34b0-40c7-41b2-82fc-ea3837f439c1" containerID="330b45b0374f544e49a26ed2438d0cdf5a605fc112db24808f22379d1a0de472" exitCode=0
Sep 18 22:39:02.314078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:02.313912    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf" event={"ID":"f73e34b0-40c7-41b2-82fc-ea3837f439c1","Type":"ContainerDied","Data":"330b45b0374f544e49a26ed2438d0cdf5a605fc112db24808f22379d1a0de472"}
Sep 18 22:39:02.314078 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:02.313950    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf" event={"ID":"f73e34b0-40c7-41b2-82fc-ea3837f439c1","Type":"ContainerStarted","Data":"c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d"}
Sep 18 22:39:03.456978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:03.456952    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:03.597954 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:03.597873    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca\") pod \"f73e34b0-40c7-41b2-82fc-ea3837f439c1\" (UID: \"f73e34b0-40c7-41b2-82fc-ea3837f439c1\") "
Sep 18 22:39:03.598302 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:03.598275    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f73e34b0-40c7-41b2-82fc-ea3837f439c1" (UID: "f73e34b0-40c7-41b2-82fc-ea3837f439c1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:39:03.699484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:03.699459    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f73e34b0-40c7-41b2-82fc-ea3837f439c1-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:04.326160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:04.326129    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"
Sep 18 22:39:04.326286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:04.326169    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf" event={"ID":"f73e34b0-40c7-41b2-82fc-ea3837f439c1","Type":"ContainerDied","Data":"c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d"}
Sep 18 22:39:04.326286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:04.326210    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c1750c824f65aa109e26c5815427c66f262a80e17ddb9cf1ef2fc26bdb494c2d"
Sep 18 22:39:16.826272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:16.826194    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"]
Sep 18 22:39:16.835084 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:16.835057    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-20e1128a-jzxhf"]
Sep 18 22:39:18.290784 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:18.290743    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f73e34b0-40c7-41b2-82fc-ea3837f439c1" path="/var/lib/kubelet/pods/f73e34b0-40c7-41b2-82fc-ea3837f439c1/volumes"
Sep 18 22:39:24.963664 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:24.963630    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:39:27.204459 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:27.204425    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:39:39.983225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:39.983155    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" podUID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" containerName="main" containerID="cri-o://f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b" gracePeriod=45
Sep 18 22:39:40.223404 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.223379    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:39:40.333689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333608    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.333689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333655    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-t4v2k\" (UniqueName: \"kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.333909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333714    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.333909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333739    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.333909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333782    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.333909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333819    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm\") pod \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\" (UID: \"df6e8028-6c60-4aba-a9c8-97b0647e8ec5\") "
Sep 18 22:39:40.334156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.333901    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:39:40.334156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.334022    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache" (OuterVolumeSpecName: "model-cache") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:39:40.334156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.334015    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home" (OuterVolumeSpecName: "home") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:39:40.334484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.334463    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.334572 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.334490    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.334572 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.334505    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.336108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.336085    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm" (OuterVolumeSpecName: "dshm") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:39:40.336108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.336088    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:39:40.336231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.336210    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k" (OuterVolumeSpecName: "kube-api-access-t4v2k") pod "df6e8028-6c60-4aba-a9c8-97b0647e8ec5" (UID: "df6e8028-6c60-4aba-a9c8-97b0647e8ec5"). InnerVolumeSpecName "kube-api-access-t4v2k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:39:40.435510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.435483    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t4v2k\" (UniqueName: \"kubernetes.io/projected/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-kube-api-access-t4v2k\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.435510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.435507    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.435510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.435516    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/df6e8028-6c60-4aba-a9c8-97b0647e8ec5-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:39:40.528907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.528872    2551 generic.go:356] "Generic (PLEG): container finished" podID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" containerID="f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b" exitCode=0
Sep 18 22:39:40.529023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.528918    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"
Sep 18 22:39:40.529023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.528957    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" event={"ID":"df6e8028-6c60-4aba-a9c8-97b0647e8ec5","Type":"ContainerDied","Data":"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"}
Sep 18 22:39:40.529023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.529005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl" event={"ID":"df6e8028-6c60-4aba-a9c8-97b0647e8ec5","Type":"ContainerDied","Data":"d03d500e032d3a305fa31b63d0a31369d7af6c846db71b20f1edaac1b0a0cd53"}
Sep 18 22:39:40.529023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.529024    2551 scope.go:117] "RemoveContainer" containerID="f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"
Sep 18 22:39:40.540149 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.540133    2551 scope.go:117] "RemoveContainer" containerID="f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"
Sep 18 22:39:40.540438 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:39:40.540418    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b\": container with ID starting with f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b not found: ID does not exist" containerID="f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"
Sep 18 22:39:40.540582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.540442    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b"} err="failed to get container status \"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b\": rpc error: code = NotFound desc = could not find container \"f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b\": container with ID starting with f9691bbf47eef62cd115454a7c260d56b1cf1c681bf58af4faa170c99e9f829b not found: ID does not exist"
Sep 18 22:39:40.566165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.566141    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:39:40.570091 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:40.570071    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-mt-20e1128a/e2e-lifecycle-model-20e1128a-kserve-6f895fdfc9-94mrl"]
Sep 18 22:39:42.292643 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:42.292613    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" path="/var/lib/kubelet/pods/df6e8028-6c60-4aba-a9c8-97b0647e8ec5/volumes"
Sep 18 22:39:57.932511 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:39:57.932477    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:40:27.215918 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:27.215875    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:40:35.454913 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.454878    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:40:35.455524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455505    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" containerName="main"
Sep 18 22:40:35.455601 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455527    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" containerName="main"
Sep 18 22:40:35.455601 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455587    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f73e34b0-40c7-41b2-82fc-ea3837f439c1" containerName="revoke-keys"
Sep 18 22:40:35.455601 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455596    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f73e34b0-40c7-41b2-82fc-ea3837f439c1" containerName="revoke-keys"
Sep 18 22:40:35.455759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455689    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f73e34b0-40c7-41b2-82fc-ea3837f439c1" containerName="revoke-keys"
Sep 18 22:40:35.455759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.455705    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="df6e8028-6c60-4aba-a9c8-97b0647e8ec5" containerName="main"
Sep 18 22:40:35.458930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.458910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.464599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.464575    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-c5fdd3\"/\"embedding-w0-c5fdd3-kserve-self-signed-certs\""
Sep 18 22:40:35.472281 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.472259    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:40:35.544232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544193    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.544384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544249    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.544384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544357    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.544482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wvtgj\" (UniqueName: \"kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.544482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544412    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.544482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.544448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645511 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645519    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645564    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645584    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wvtgj\" (UniqueName: \"kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645603    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645704 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645627    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645909    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.645979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645943    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.646041 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.645995    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.647807 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.647784    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.648195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.648177    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.653365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.653345    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wvtgj\" (UniqueName: \"kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj\") pod \"embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") " pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:35.771272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:35.771221    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:36.071856 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:36.071831    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:40:36.074296 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:40:36.074267    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod463a644f_4608_46c0_8c95_2e6a315d5d22.slice/crio-1757135fd35c8273fc60b502100022eded3a5d11c4736bb9068c86fff730cd63 WatchSource:0}: Error finding container 1757135fd35c8273fc60b502100022eded3a5d11c4736bb9068c86fff730cd63: Status 404 returned error can't find the container with id 1757135fd35c8273fc60b502100022eded3a5d11c4736bb9068c86fff730cd63
Sep 18 22:40:36.853691 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:36.853658    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" event={"ID":"463a644f-4608-46c0-8c95-2e6a315d5d22","Type":"ContainerStarted","Data":"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"}
Sep 18 22:40:36.854140 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:36.853698    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" event={"ID":"463a644f-4608-46c0-8c95-2e6a315d5d22","Type":"ContainerStarted","Data":"1757135fd35c8273fc60b502100022eded3a5d11c4736bb9068c86fff730cd63"}
Sep 18 22:40:36.875116 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:36.875053    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" podStartSLOduration=1.875017573 podStartE2EDuration="1.875017573s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:40:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:40:36.873596427 +0000 UTC m=+1429.055180974" watchObservedRunningTime="2026-09-18 22:40:36.875017573 +0000 UTC m=+1429.056602121"
Sep 18 22:40:45.771826 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:45.771741    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:45.772310 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:45.771826    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:45.784570 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:45.784549    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:45.913439 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:45.913420    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:40:48.507107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:48.507078    2551 scope.go:117] "RemoveContainer" containerID="20f63baa5fd65b3f3b77f1d4cd6e9f77d136f292d67fa1c3f67da5ae11dc54ee"
Sep 18 22:40:48.516126 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:40:48.516108    2551 scope.go:117] "RemoveContainer" containerID="f51ac3c73544c60395a6382c05fed283ae7aedd0e50a135413dcfbff43dc7aa5"
Sep 18 22:41:03.012618 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:03.012581    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:41:36.186212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:36.186179    2551 generic.go:356] "Generic (PLEG): container finished" podID="18261b97-897d-4a5d-b471-75099643a20f" containerID="3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe" exitCode=137
Sep 18 22:41:36.186600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:36.186249    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerDied","Data":"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"}
Sep 18 22:41:36.186600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:36.186575    2551 scope.go:117] "RemoveContainer" containerID="3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"
Sep 18 22:41:37.194328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:37.194294    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerStarted","Data":"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"}
Sep 18 22:41:37.194806 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:37.194514    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:41:48.201947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:48.201909    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:41:48.354579 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:48.354550    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:41:48.355312 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:48.355294    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:41:59.003119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.003084    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:41:59.006509 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.006491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.015169 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.015147    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-d90856\""
Sep 18 22:41:59.015302 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.015197    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-d90856-dockercfg-5t2zq\""
Sep 18 22:41:59.030479 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.030458    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:41:59.130802 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.130775    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.130924 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.130818    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s47gw\" (UniqueName: \"kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.232182 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.232137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.232372 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.232210    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s47gw\" (UniqueName: \"kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.232827 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.232806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.241813 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.241782    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s47gw\" (UniqueName: \"kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw\") pod \"payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") " pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.316864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.316792    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:41:59.600726 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:41:59.600693    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:41:59.602586 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:41:59.602555    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod23be1b41_2297_478d_8e01_dbc8972f379b.slice/crio-0f46634213bcffd7fbd6b83c8de2fbf495a09fcab864205327228a4491c2aa92 WatchSource:0}: Error finding container 0f46634213bcffd7fbd6b83c8de2fbf495a09fcab864205327228a4491c2aa92: Status 404 returned error can't find the container with id 0f46634213bcffd7fbd6b83c8de2fbf495a09fcab864205327228a4491c2aa92
Sep 18 22:42:00.330909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:00.330868    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" event={"ID":"23be1b41-2297-478d-8e01-dbc8972f379b","Type":"ContainerStarted","Data":"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"}
Sep 18 22:42:00.331382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:00.331012    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:42:00.331382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:00.331070    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" event={"ID":"23be1b41-2297-478d-8e01-dbc8972f379b","Type":"ContainerStarted","Data":"0f46634213bcffd7fbd6b83c8de2fbf495a09fcab864205327228a4491c2aa92"}
Sep 18 22:42:00.351637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:00.351595    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" podStartSLOduration=2.351582315 podStartE2EDuration="2.351582315s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:41:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:00.35033898 +0000 UTC m=+1512.531923550" watchObservedRunningTime="2026-09-18 22:42:00.351582315 +0000 UTC m=+1512.533166859"
Sep 18 22:42:03.862146 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:03.862099    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:42:11.343137 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:11.343057    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:42:12.831850 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.831811    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"]
Sep 18 22:42:12.836638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.836611    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:12.839543 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.839510    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:42:12.841276 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.841252    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"]
Sep 18 22:42:12.858090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.858068    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl\" (UID: \"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:12.958933 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.958900    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl\" (UID: \"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:12.959602 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:12.959582    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl\" (UID: \"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:13.152464 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:13.152440    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:13.425318 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:13.425293    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"]
Sep 18 22:42:13.427128 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:42:13.427094    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53c3a4fd_4918_4d62_b2c8_c5e9faa477d9.slice/crio-add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda WatchSource:0}: Error finding container add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda: Status 404 returned error can't find the container with id add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda
Sep 18 22:42:14.420134 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:14.420091    2551 generic.go:356] "Generic (PLEG): container finished" podID="53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" containerID="059bc860b7ac50fdc066bb92b46d98da1fd2bf6bd1c284b30c79f6761f78e450" exitCode=0
Sep 18 22:42:14.420515 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:14.420159    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl" event={"ID":"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9","Type":"ContainerDied","Data":"059bc860b7ac50fdc066bb92b46d98da1fd2bf6bd1c284b30c79f6761f78e450"}
Sep 18 22:42:14.420515 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:14.420201    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl" event={"ID":"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9","Type":"ContainerStarted","Data":"add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda"}
Sep 18 22:42:15.569539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:15.569514    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:15.686262 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:15.686181    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca\") pod \"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9\" (UID: \"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9\") "
Sep 18 22:42:15.686523 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:15.686493    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" (UID: "53c3a4fd-4918-4d62-b2c8-c5e9faa477d9"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:15.786984 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:15.786953    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:16.438540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:16.438444    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl" event={"ID":"53c3a4fd-4918-4d62-b2c8-c5e9faa477d9","Type":"ContainerDied","Data":"add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda"}
Sep 18 22:42:16.438540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:16.438495    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"
Sep 18 22:42:16.438540 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:16.438537    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="add907ea500323017d8bc7b71f7a8f93360e9c9e0c5b8d51d0f29c53d5f83dda"
Sep 18 22:42:17.792059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:17.792000    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:42:17.801719 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:17.801693    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:42:17.810947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:17.810922    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:42:17.824628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:17.824597    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:42:24.405808 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.405763    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"]
Sep 18 22:42:24.406461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.406440    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" containerName="revoke-keys"
Sep 18 22:42:24.406550 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.406464    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" containerName="revoke-keys"
Sep 18 22:42:24.406607 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.406583    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" containerName="revoke-keys"
Sep 18 22:42:24.409687 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.409666    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:24.412196 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.412175    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:42:24.417751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.417731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"]
Sep 18 22:42:24.468899 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.468872    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9\" (UID: \"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:24.569871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.569844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9\" (UID: \"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:24.570461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.570443    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9\" (UID: \"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:24.721462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.721392    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:24.996196 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:24.996167    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"]
Sep 18 22:42:24.998507 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:42:24.998483    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod214cfe84_4a60_46dc_83e3_26a8eaf7fdd5.slice/crio-725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a WatchSource:0}: Error finding container 725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a: Status 404 returned error can't find the container with id 725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a
Sep 18 22:42:25.490009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:25.489977    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9" event={"ID":"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5","Type":"ContainerStarted","Data":"b0920a6b170351d6a6d31a77d91b0559e2c60f0adf6b205af6b2fa9d9c266369"}
Sep 18 22:42:25.490376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:25.490044    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9" event={"ID":"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5","Type":"ContainerStarted","Data":"725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a"}
Sep 18 22:42:25.505869 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:25.505703    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9" podStartSLOduration=1.505689341 podStartE2EDuration="1.505689341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:25.50525746 +0000 UTC m=+1537.686842006" watchObservedRunningTime="2026-09-18 22:42:25.505689341 +0000 UTC m=+1537.687273887"
Sep 18 22:42:26.497185 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:26.497147    2551 generic.go:356] "Generic (PLEG): container finished" podID="214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" containerID="b0920a6b170351d6a6d31a77d91b0559e2c60f0adf6b205af6b2fa9d9c266369" exitCode=0
Sep 18 22:42:26.497549 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:26.497226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9" event={"ID":"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5","Type":"ContainerDied","Data":"b0920a6b170351d6a6d31a77d91b0559e2c60f0adf6b205af6b2fa9d9c266369"}
Sep 18 22:42:27.644297 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:27.644272    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:27.703300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:27.703272    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca\") pod \"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5\" (UID: \"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5\") "
Sep 18 22:42:27.703595 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:27.703572    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" (UID: "214cfe84-4a60-46dc-83e3-26a8eaf7fdd5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:27.804478 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:27.804415    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:28.512276 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:28.512249    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"
Sep 18 22:42:28.512444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:28.512289    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9" event={"ID":"214cfe84-4a60-46dc-83e3-26a8eaf7fdd5","Type":"ContainerDied","Data":"725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a"}
Sep 18 22:42:28.512444 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:28.512337    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="725dad5b556ed4f4a738a3e47d77556ade950d9ec2bd37a164ad66bd579f477a"
Sep 18 22:42:32.811942 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:32.811898    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" podUID="88a218fc-8465-4a4d-9520-7cbc7e195d99" containerName="main" containerID="cri-o://292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3" gracePeriod=45
Sep 18 22:42:32.820493 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:32.820463    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" podUID="463a644f-4608-46c0-8c95-2e6a315d5d22" containerName="main" containerID="cri-o://ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5" gracePeriod=45
Sep 18 22:42:32.829694 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:32.829665    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" podUID="70986b94-5bd0-4915-82ab-146327e59ba8" containerName="main" containerID="cri-o://a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7" gracePeriod=45
Sep 18 22:42:32.844072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:32.844008    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" podUID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" containerName="main" containerID="cri-o://657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52" gracePeriod=45
Sep 18 22:42:32.912532 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:42:32.912502    2551 cadvisor_stats_provider.go:567] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70986b94_5bd0_4915_82ab_146327e59ba8.slice/crio-a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod70986b94_5bd0_4915_82ab_146327e59ba8.slice/crio-conmon-a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7.scope\": RecentStats: unable to find data in memory cache]"
Sep 18 22:42:33.146089 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.146057    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:42:33.165495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.165471    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:42:33.169699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.169679    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:42:33.174038 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.174008    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:42:33.260831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.260793    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9xkcf\" (UniqueName: \"kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261025 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.260873    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261048    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.261124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261102    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261108    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.261288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261129    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261157    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261188    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261238    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.261288 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261273    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261302    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261351    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261383    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261422    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261456    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gwcst\" (UniqueName: \"kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261468    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.261525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261486    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261553    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261589    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261616    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261654    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home\") pod \"88a218fc-8465-4a4d-9520-7cbc7e195d99\" (UID: \"88a218fc-8465-4a4d-9520-7cbc7e195d99\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261681    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache" (OuterVolumeSpecName: "model-cache") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261734    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n229q\" (UniqueName: \"kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q\") pod \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\" (UID: \"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261781    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home\") pod \"70986b94-5bd0-4915-82ab-146327e59ba8\" (UID: \"70986b94-5bd0-4915-82ab-146327e59ba8\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261812    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.261889 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.261847    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wvtgj\" (UniqueName: \"kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj\") pod \"463a644f-4608-46c0-8c95-2e6a315d5d22\" (UID: \"463a644f-4608-46c0-8c95-2e6a315d5d22\") "
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262070    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home" (OuterVolumeSpecName: "home") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262351    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home" (OuterVolumeSpecName: "home") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262393    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262412    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262427    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.262465 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.262443    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.263709 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.263681    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:33.263831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.263753    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf" (OuterVolumeSpecName: "kube-api-access-9xkcf") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "kube-api-access-9xkcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:33.263930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.263894    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home" (OuterVolumeSpecName: "home") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.263998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.263950    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.263998 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.263945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache" (OuterVolumeSpecName: "model-cache") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.265095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.264302    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.265095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.264606    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home" (OuterVolumeSpecName: "home") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.265095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.264790    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm" (OuterVolumeSpecName: "dshm") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.265095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.264803    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj" (OuterVolumeSpecName: "kube-api-access-wvtgj") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "kube-api-access-wvtgj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:33.265095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.264827    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache" (OuterVolumeSpecName: "model-cache") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.266393 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.266372    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:33.266552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.266531    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache" (OuterVolumeSpecName: "model-cache") pod "463a644f-4608-46c0-8c95-2e6a315d5d22" (UID: "463a644f-4608-46c0-8c95-2e6a315d5d22"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.266960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.266935    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:33.267270 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.267235    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm" (OuterVolumeSpecName: "dshm") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.267610 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.267575    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst" (OuterVolumeSpecName: "kube-api-access-gwcst") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "kube-api-access-gwcst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:33.268396 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.268368    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm" (OuterVolumeSpecName: "dshm") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.268498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.268481    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm" (OuterVolumeSpecName: "dshm") pod "88a218fc-8465-4a4d-9520-7cbc7e195d99" (UID: "88a218fc-8465-4a4d-9520-7cbc7e195d99"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:42:33.268731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.268712    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "70986b94-5bd0-4915-82ab-146327e59ba8" (UID: "70986b94-5bd0-4915-82ab-146327e59ba8"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:42:33.270020 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.269998    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q" (OuterVolumeSpecName: "kube-api-access-n229q") pod "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" (UID: "5b57a4a7-e7a6-4d87-90d5-cd011aa46c43"). InnerVolumeSpecName "kube-api-access-n229q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:33.306073 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.306048    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:42:33.306348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.306296    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" podUID="4e091709-cf96-42ed-8e91-43bd483c8f62" containerName="payload-processing" containerID="cri-o://07747f3e5caf1201ca1f958f68415f69a5e6ebe807c2c912043b9cfc2b577726" gracePeriod=30
Sep 18 22:42:33.363711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363672    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wvtgj\" (UniqueName: \"kubernetes.io/projected/463a644f-4608-46c0-8c95-2e6a315d5d22-kube-api-access-wvtgj\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363711 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363708    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9xkcf\" (UniqueName: \"kubernetes.io/projected/88a218fc-8465-4a4d-9520-7cbc7e195d99-kube-api-access-9xkcf\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363721    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/463a644f-4608-46c0-8c95-2e6a315d5d22-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363735    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363749    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363759    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363770    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363782    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363794    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/70986b94-5bd0-4915-82ab-146327e59ba8-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363805    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363817    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363827    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gwcst\" (UniqueName: \"kubernetes.io/projected/70986b94-5bd0-4915-82ab-146327e59ba8-kube-api-access-gwcst\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363839    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/70986b94-5bd0-4915-82ab-146327e59ba8-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363851    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.363864 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363863    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.364269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363874    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/88a218fc-8465-4a4d-9520-7cbc7e195d99-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.364269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363887    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/88a218fc-8465-4a4d-9520-7cbc7e195d99-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.364269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363898    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.364269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363910    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n229q\" (UniqueName: \"kubernetes.io/projected/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43-kube-api-access-n229q\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.364269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.363922    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/463a644f-4608-46c0-8c95-2e6a315d5d22-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.494328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.494298    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"]
Sep 18 22:42:33.504593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.504566    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-c5fdd3-shnwl"]
Sep 18 22:42:33.545044 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.545003    2551 generic.go:356] "Generic (PLEG): container finished" podID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" containerID="657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52" exitCode=0
Sep 18 22:42:33.545179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.545079    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" event={"ID":"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43","Type":"ContainerDied","Data":"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"}
Sep 18 22:42:33.545179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.545112    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"
Sep 18 22:42:33.545179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.545126    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6" event={"ID":"5b57a4a7-e7a6-4d87-90d5-cd011aa46c43","Type":"ContainerDied","Data":"55cdc1b54e27e6c164cdb0321e92e253dfd7e0ad8001f679f4aebf361c2fef50"}
Sep 18 22:42:33.545179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.545150    2551 scope.go:117] "RemoveContainer" containerID="657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"
Sep 18 22:42:33.546568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.546547    2551 generic.go:356] "Generic (PLEG): container finished" podID="4e091709-cf96-42ed-8e91-43bd483c8f62" containerID="07747f3e5caf1201ca1f958f68415f69a5e6ebe807c2c912043b9cfc2b577726" exitCode=0
Sep 18 22:42:33.546670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.546607    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" event={"ID":"4e091709-cf96-42ed-8e91-43bd483c8f62","Type":"ContainerDied","Data":"07747f3e5caf1201ca1f958f68415f69a5e6ebe807c2c912043b9cfc2b577726"}
Sep 18 22:42:33.548245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.548222    2551 generic.go:356] "Generic (PLEG): container finished" podID="463a644f-4608-46c0-8c95-2e6a315d5d22" containerID="ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5" exitCode=0
Sep 18 22:42:33.548348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.548273    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"
Sep 18 22:42:33.548348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.548312    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" event={"ID":"463a644f-4608-46c0-8c95-2e6a315d5d22","Type":"ContainerDied","Data":"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"}
Sep 18 22:42:33.548482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.548347    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf" event={"ID":"463a644f-4608-46c0-8c95-2e6a315d5d22","Type":"ContainerDied","Data":"1757135fd35c8273fc60b502100022eded3a5d11c4736bb9068c86fff730cd63"}
Sep 18 22:42:33.548758 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.548741    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:42:33.549866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.549847    2551 generic.go:356] "Generic (PLEG): container finished" podID="70986b94-5bd0-4915-82ab-146327e59ba8" containerID="a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7" exitCode=0
Sep 18 22:42:33.549959 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.549896    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"
Sep 18 22:42:33.549959 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.549933    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" event={"ID":"70986b94-5bd0-4915-82ab-146327e59ba8","Type":"ContainerDied","Data":"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"}
Sep 18 22:42:33.550113 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.549963    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr" event={"ID":"70986b94-5bd0-4915-82ab-146327e59ba8","Type":"ContainerDied","Data":"fff1eca6ff21722c0cd4a400bb908af8dbfe9a0c51b4fcf5a5512cfdb7a90abc"}
Sep 18 22:42:33.551759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.551729    2551 generic.go:356] "Generic (PLEG): container finished" podID="88a218fc-8465-4a4d-9520-7cbc7e195d99" containerID="292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3" exitCode=0
Sep 18 22:42:33.551919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.551762    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" event={"ID":"88a218fc-8465-4a4d-9520-7cbc7e195d99","Type":"ContainerDied","Data":"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"}
Sep 18 22:42:33.551919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.551771    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"
Sep 18 22:42:33.551919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.551786    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7" event={"ID":"88a218fc-8465-4a4d-9520-7cbc7e195d99","Type":"ContainerDied","Data":"a208ce585ec4b582b9c728b3f4fb41f12a24b0bd77f2ecb1567f3f3366171e53"}
Sep 18 22:42:33.560498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.559135    2551 scope.go:117] "RemoveContainer" containerID="657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"
Sep 18 22:42:33.561392 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:42:33.561261    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52\": container with ID starting with 657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52 not found: ID does not exist" containerID="657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"
Sep 18 22:42:33.561392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.561297    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52"} err="failed to get container status \"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52\": rpc error: code = NotFound desc = could not find container \"657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52\": container with ID starting with 657b80973ac5fd27b334290e6b40b33290d2dff51b18bdfc4e0bef1554ba2a52 not found: ID does not exist"
Sep 18 22:42:33.561392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.561319    2551 scope.go:117] "RemoveContainer" containerID="ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"
Sep 18 22:42:33.575153 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.575137    2551 scope.go:117] "RemoveContainer" containerID="ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"
Sep 18 22:42:33.577469 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:42:33.577378    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5\": container with ID starting with ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5 not found: ID does not exist" containerID="ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"
Sep 18 22:42:33.577469 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.577413    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5"} err="failed to get container status \"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5\": rpc error: code = NotFound desc = could not find container \"ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5\": container with ID starting with ab5c9fa2d0275dcc12a910ef07898a0eef8c946990b1f5dda9dcdf66337676f5 not found: ID does not exist"
Sep 18 22:42:33.577469 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.577435    2551 scope.go:117] "RemoveContainer" containerID="a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"
Sep 18 22:42:33.587153 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.587125    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:42:33.588279 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.588247    2551 scope.go:117] "RemoveContainer" containerID="a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"
Sep 18 22:42:33.588638 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:42:33.588607    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7\": container with ID starting with a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7 not found: ID does not exist" containerID="a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"
Sep 18 22:42:33.588748 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.588648    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7"} err="failed to get container status \"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7\": rpc error: code = NotFound desc = could not find container \"a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7\": container with ID starting with a667f5da7ff7b4277ee60179cc0210c37d083c5978c4a33ecaa5c35d26c6dbb7 not found: ID does not exist"
Sep 18 22:42:33.588748 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.588670    2551 scope.go:117] "RemoveContainer" containerID="292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"
Sep 18 22:42:33.594525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.594501    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/embedding-w0-c5fdd3-kserve-5c7cfdc98f-j4vgf"]
Sep 18 22:42:33.601127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.601108    2551 scope.go:117] "RemoveContainer" containerID="292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"
Sep 18 22:42:33.601457 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:42:33.601436    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3\": container with ID starting with 292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3 not found: ID does not exist" containerID="292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"
Sep 18 22:42:33.601511 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.601459    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3"} err="failed to get container status \"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3\": rpc error: code = NotFound desc = could not find container \"292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3\": container with ID starting with 292020b23b62e98b62a7a9d6cc40f0ae80d3c9c27dd6097bebea5665f5ed84f3 not found: ID does not exist"
Sep 18 22:42:33.604376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.604356    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:42:33.610943 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.610923    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/distinct-w0-c5fdd3-kserve-676dd4d9b-gtwb7"]
Sep 18 22:42:33.622174 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.622151    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:42:33.629420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.629400    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/simulator-w0-c5fdd3-kserve-7879f97864-5vxs6"]
Sep 18 22:42:33.641579 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.641558    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:42:33.646207 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.646180    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-c5fdd3/premium-w0-c5fdd3-kserve-86c7f8c8d5-5gxwr"]
Sep 18 22:42:33.667693 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.667674    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4lh82\" (UniqueName: \"kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82\") pod \"4e091709-cf96-42ed-8e91-43bd483c8f62\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") "
Sep 18 22:42:33.667774 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.667739    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume\") pod \"4e091709-cf96-42ed-8e91-43bd483c8f62\" (UID: \"4e091709-cf96-42ed-8e91-43bd483c8f62\") "
Sep 18 22:42:33.668123 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.668100    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4e091709-cf96-42ed-8e91-43bd483c8f62" (UID: "4e091709-cf96-42ed-8e91-43bd483c8f62"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:33.669645 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.669626    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82" (OuterVolumeSpecName: "kube-api-access-4lh82") pod "4e091709-cf96-42ed-8e91-43bd483c8f62" (UID: "4e091709-cf96-42ed-8e91-43bd483c8f62"). InnerVolumeSpecName "kube-api-access-4lh82". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:42:33.769400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.769378    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4e091709-cf96-42ed-8e91-43bd483c8f62-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:33.769400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:33.769397    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lh82\" (UniqueName: \"kubernetes.io/projected/4e091709-cf96-42ed-8e91-43bd483c8f62-kube-api-access-4lh82\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:34.290916 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.290880    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="463a644f-4608-46c0-8c95-2e6a315d5d22" path="/var/lib/kubelet/pods/463a644f-4608-46c0-8c95-2e6a315d5d22/volumes"
Sep 18 22:42:34.291433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.291372    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="53c3a4fd-4918-4d62-b2c8-c5e9faa477d9" path="/var/lib/kubelet/pods/53c3a4fd-4918-4d62-b2c8-c5e9faa477d9/volumes"
Sep 18 22:42:34.291804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.291781    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" path="/var/lib/kubelet/pods/5b57a4a7-e7a6-4d87-90d5-cd011aa46c43/volumes"
Sep 18 22:42:34.292518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.292495    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="70986b94-5bd0-4915-82ab-146327e59ba8" path="/var/lib/kubelet/pods/70986b94-5bd0-4915-82ab-146327e59ba8/volumes"
Sep 18 22:42:34.292927 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.292909    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="88a218fc-8465-4a4d-9520-7cbc7e195d99" path="/var/lib/kubelet/pods/88a218fc-8465-4a4d-9520-7cbc7e195d99/volumes"
Sep 18 22:42:34.560285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.560203    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"
Sep 18 22:42:34.560285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.560199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq" event={"ID":"4e091709-cf96-42ed-8e91-43bd483c8f62","Type":"ContainerDied","Data":"3ae7ebe39424afc719c272cc6709e90c88bfa253302ae122655fa3da759e6909"}
Sep 18 22:42:34.560490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.560329    2551 scope.go:117] "RemoveContainer" containerID="07747f3e5caf1201ca1f958f68415f69a5e6ebe807c2c912043b9cfc2b577726"
Sep 18 22:42:34.576819 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.576795    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:42:34.581436 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:34.581415    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w0-c5fdd3-54bbb687cc-8hcbq"]
Sep 18 22:42:36.291103 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:36.291064    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4e091709-cf96-42ed-8e91-43bd483c8f62" path="/var/lib/kubelet/pods/4e091709-cf96-42ed-8e91-43bd483c8f62/volumes"
Sep 18 22:42:43.899072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:43.899019    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"]
Sep 18 22:42:43.900953 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:43.900930    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-37cf0f-4bhv9"]
Sep 18 22:42:44.012619 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:44.012578    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:42:44.291367 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:44.291293    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" path="/var/lib/kubelet/pods/214cfe84-4a60-46dc-83e3-26a8eaf7fdd5/volumes"
Sep 18 22:42:48.550951 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:48.550921    2551 scope.go:117] "RemoveContainer" containerID="6566968b01518f732ee3795ebd0e85fc660a7c0fb6cfdcf5b959a18841dbadf5"
Sep 18 22:42:48.559450 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:48.559432    2551 scope.go:117] "RemoveContainer" containerID="41eb53e0de0caaa8446c573730f3fc8f3654611792555a19c60c8ac3babe7b59"
Sep 18 22:42:51.574805 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.574769    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"]
Sep 18 22:42:51.575245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575231    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="88a218fc-8465-4a4d-9520-7cbc7e195d99" containerName="main"
Sep 18 22:42:51.575245 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575246    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="88a218fc-8465-4a4d-9520-7cbc7e195d99" containerName="main"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575257    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="70986b94-5bd0-4915-82ab-146327e59ba8" containerName="main"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575266    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="70986b94-5bd0-4915-82ab-146327e59ba8" containerName="main"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575285    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="463a644f-4608-46c0-8c95-2e6a315d5d22" containerName="main"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575290    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="463a644f-4608-46c0-8c95-2e6a315d5d22" containerName="main"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575302    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" containerName="revoke-keys"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575307    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" containerName="revoke-keys"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575314    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4e091709-cf96-42ed-8e91-43bd483c8f62" containerName="payload-processing"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575319    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4e091709-cf96-42ed-8e91-43bd483c8f62" containerName="payload-processing"
Sep 18 22:42:51.575334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575332    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575338    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575394    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="88a218fc-8465-4a4d-9520-7cbc7e195d99" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575403    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5b57a4a7-e7a6-4d87-90d5-cd011aa46c43" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575410    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="463a644f-4608-46c0-8c95-2e6a315d5d22" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575418    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="70986b94-5bd0-4915-82ab-146327e59ba8" containerName="main"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575427    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4e091709-cf96-42ed-8e91-43bd483c8f62" containerName="payload-processing"
Sep 18 22:42:51.575593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.575436    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="214cfe84-4a60-46dc-83e3-26a8eaf7fdd5" containerName="revoke-keys"
Sep 18 22:42:51.578269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.578250    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:51.580885 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.580865    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:42:51.586237 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.586215    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"]
Sep 18 22:42:51.631305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.631279    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d90856-79flz\" (UID: \"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:51.732423 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.732394    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d90856-79flz\" (UID: \"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:51.733008 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.732983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-d90856-79flz\" (UID: \"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:51.889690 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:51.889633    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:52.202930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:52.202900    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"]
Sep 18 22:42:52.203767 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:42:52.203738    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0ea708e8_f3cc_4c11_ac23_cfaa353c45fa.slice/crio-3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2 WatchSource:0}: Error finding container 3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2: Status 404 returned error can't find the container with id 3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2
Sep 18 22:42:52.673791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:52.673748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz" event={"ID":"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa","Type":"ContainerStarted","Data":"972040a58b263c50062729b2978a80b72338a2502857e59ec7139371a2cd10ae"}
Sep 18 22:42:52.674121 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:52.673811    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz" event={"ID":"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa","Type":"ContainerStarted","Data":"3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2"}
Sep 18 22:42:52.691199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:52.691152    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz" podStartSLOduration=1.6911364340000001 podStartE2EDuration="1.691136434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:42:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:42:52.688829748 +0000 UTC m=+1564.870414294" watchObservedRunningTime="2026-09-18 22:42:52.691136434 +0000 UTC m=+1564.872720973"
Sep 18 22:42:53.680649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:53.680608    2551 generic.go:356] "Generic (PLEG): container finished" podID="0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" containerID="972040a58b263c50062729b2978a80b72338a2502857e59ec7139371a2cd10ae" exitCode=0
Sep 18 22:42:53.680987 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:53.680700    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz" event={"ID":"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa","Type":"ContainerDied","Data":"972040a58b263c50062729b2978a80b72338a2502857e59ec7139371a2cd10ae"}
Sep 18 22:42:54.814014 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:54.813985    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:42:54.862474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:54.862444    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca\") pod \"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa\" (UID: \"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa\") "
Sep 18 22:42:54.862759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:54.862737    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" (UID: "0ea708e8-f3cc-4c11-ac23-cfaa353c45fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:42:54.963433 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:54.963381    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:42:55.693290 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:55.693256    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz" event={"ID":"0ea708e8-f3cc-4c11-ac23-cfaa353c45fa","Type":"ContainerDied","Data":"3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2"}
Sep 18 22:42:55.693529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:55.693303    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fed5f8b09a2ac4e57d4ac7913fe28655f0f309919467453d1a6048693ddecc2"
Sep 18 22:42:55.693529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:42:55.693270    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"
Sep 18 22:43:07.164984 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.164949    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:43:07.165452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.165196    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" podUID="23be1b41-2297-478d-8e01-dbc8972f379b" containerName="payload-processing" containerID="cri-o://e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84" gracePeriod=30
Sep 18 22:43:07.404314 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.404279    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"]
Sep 18 22:43:07.407566 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.407547    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-d90856-79flz"]
Sep 18 22:43:07.429538 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.429482    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:43:07.573380 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.573353    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-s47gw\" (UniqueName: \"kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw\") pod \"23be1b41-2297-478d-8e01-dbc8972f379b\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") "
Sep 18 22:43:07.573561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.573465    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume\") pod \"23be1b41-2297-478d-8e01-dbc8972f379b\" (UID: \"23be1b41-2297-478d-8e01-dbc8972f379b\") "
Sep 18 22:43:07.573857 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.573835    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "23be1b41-2297-478d-8e01-dbc8972f379b" (UID: "23be1b41-2297-478d-8e01-dbc8972f379b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:07.575672 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.575646    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw" (OuterVolumeSpecName: "kube-api-access-s47gw") pod "23be1b41-2297-478d-8e01-dbc8972f379b" (UID: "23be1b41-2297-478d-8e01-dbc8972f379b"). InnerVolumeSpecName "kube-api-access-s47gw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:07.674979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.674943    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/23be1b41-2297-478d-8e01-dbc8972f379b-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:07.674979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.674974    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s47gw\" (UniqueName: \"kubernetes.io/projected/23be1b41-2297-478d-8e01-dbc8972f379b-kube-api-access-s47gw\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:07.760970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.760907    2551 generic.go:356] "Generic (PLEG): container finished" podID="23be1b41-2297-478d-8e01-dbc8972f379b" containerID="e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84" exitCode=0
Sep 18 22:43:07.760970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.760954    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"
Sep 18 22:43:07.761134 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.760970    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" event={"ID":"23be1b41-2297-478d-8e01-dbc8972f379b","Type":"ContainerDied","Data":"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"}
Sep 18 22:43:07.761134 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.761017    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z" event={"ID":"23be1b41-2297-478d-8e01-dbc8972f379b","Type":"ContainerDied","Data":"0f46634213bcffd7fbd6b83c8de2fbf495a09fcab864205327228a4491c2aa92"}
Sep 18 22:43:07.761134 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.761058    2551 scope.go:117] "RemoveContainer" containerID="e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"
Sep 18 22:43:07.770853 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.770825    2551 scope.go:117] "RemoveContainer" containerID="e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"
Sep 18 22:43:07.771171 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:07.771152    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84\": container with ID starting with e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84 not found: ID does not exist" containerID="e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"
Sep 18 22:43:07.771263 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.771178    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84"} err="failed to get container status \"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84\": rpc error: code = NotFound desc = could not find container \"e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84\": container with ID starting with e144d4ab57af9df0755099a7b5614a38e7a7b8b0c66218694c26e386fd9ceb84 not found: ID does not exist"
Sep 18 22:43:07.785881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.785862    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:43:07.787757 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:07.787740    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-d90856-6ccd4dbdbf-spf7z"]
Sep 18 22:43:08.291160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:08.291121    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" path="/var/lib/kubelet/pods/0ea708e8-f3cc-4c11-ac23-cfaa353c45fa/volumes"
Sep 18 22:43:08.291568 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:08.291554    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="23be1b41-2297-478d-8e01-dbc8972f379b" path="/var/lib/kubelet/pods/23be1b41-2297-478d-8e01-dbc8972f379b/volumes"
Sep 18 22:43:14.248085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248049    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"]
Sep 18 22:43:14.248456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248446    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" containerName="revoke-keys"
Sep 18 22:43:14.248456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248456    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" containerName="revoke-keys"
Sep 18 22:43:14.248530 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248472    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="23be1b41-2297-478d-8e01-dbc8972f379b" containerName="payload-processing"
Sep 18 22:43:14.248530 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248477    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="23be1b41-2297-478d-8e01-dbc8972f379b" containerName="payload-processing"
Sep 18 22:43:14.248914 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.248888    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0ea708e8-f3cc-4c11-ac23-cfaa353c45fa" containerName="revoke-keys"
Sep 18 22:43:14.251570 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.249507    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="23be1b41-2297-478d-8e01-dbc8972f379b" containerName="payload-processing"
Sep 18 22:43:14.257592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.257564    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:14.260101 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.260072    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"]
Sep 18 22:43:14.260219 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.260142    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:43:14.332100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.332073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd\" (UID: \"364b5f84-3526-42a4-b6ac-89a19400e335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:14.433590 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.433565    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd\" (UID: \"364b5f84-3526-42a4-b6ac-89a19400e335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:14.434118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.434102    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd\" (UID: \"364b5f84-3526-42a4-b6ac-89a19400e335\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:14.571791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.571730    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:14.861316 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:14.861288    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"]
Sep 18 22:43:14.862997 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:43:14.862970    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod364b5f84_3526_42a4_b6ac_89a19400e335.slice/crio-01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081 WatchSource:0}: Error finding container 01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081: Status 404 returned error can't find the container with id 01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081
Sep 18 22:43:15.810124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:15.810055    2551 generic.go:356] "Generic (PLEG): container finished" podID="364b5f84-3526-42a4-b6ac-89a19400e335" containerID="7c1d2cb183a45e52adbb83e37521a670498c2fdea00d2a7d05c31848b22454fb" exitCode=0
Sep 18 22:43:15.810476 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:15.810120    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd" event={"ID":"364b5f84-3526-42a4-b6ac-89a19400e335","Type":"ContainerDied","Data":"7c1d2cb183a45e52adbb83e37521a670498c2fdea00d2a7d05c31848b22454fb"}
Sep 18 22:43:15.810476 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:15.810183    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd" event={"ID":"364b5f84-3526-42a4-b6ac-89a19400e335","Type":"ContainerStarted","Data":"01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081"}
Sep 18 22:43:16.958800 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:16.958773    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:17.057213 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.057179    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca\") pod \"364b5f84-3526-42a4-b6ac-89a19400e335\" (UID: \"364b5f84-3526-42a4-b6ac-89a19400e335\") "
Sep 18 22:43:17.057575 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.057547    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "364b5f84-3526-42a4-b6ac-89a19400e335" (UID: "364b5f84-3526-42a4-b6ac-89a19400e335"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:17.158213 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.158182    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/364b5f84-3526-42a4-b6ac-89a19400e335-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:17.823934 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.823907    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"
Sep 18 22:43:17.823934 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.823919    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd" event={"ID":"364b5f84-3526-42a4-b6ac-89a19400e335","Type":"ContainerDied","Data":"01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081"}
Sep 18 22:43:17.824176 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:17.823960    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="01b0237e74369916f66445c744ac4b289883137dc97c5a48d7d406d4abb6c081"
Sep 18 22:43:20.093412 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:20.093376    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:43:24.924892 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:24.924853    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"]
Sep 18 22:43:24.926978 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:24.926950    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-8faaa1-ngjjd"]
Sep 18 22:43:26.290773 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:26.290732    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="364b5f84-3526-42a4-b6ac-89a19400e335" path="/var/lib/kubelet/pods/364b5f84-3526-42a4-b6ac-89a19400e335/volumes"
Sep 18 22:43:31.647947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.647912    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:43:31.648434 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.648142    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy" containerID="cri-o://f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220" gracePeriod=30
Sep 18 22:43:31.985095 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.984991    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"]
Sep 18 22:43:31.985486 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.985468    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="364b5f84-3526-42a4-b6ac-89a19400e335" containerName="revoke-keys"
Sep 18 22:43:31.985580 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.985489    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="364b5f84-3526-42a4-b6ac-89a19400e335" containerName="revoke-keys"
Sep 18 22:43:31.985637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.985617    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="364b5f84-3526-42a4-b6ac-89a19400e335" containerName="revoke-keys"
Sep 18 22:43:31.988961 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.988939    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:31.991305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.991285    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:43:31.999561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:31.999537    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"]
Sep 18 22:43:32.089791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.089767    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4\" (UID: \"e005cb9b-d9a9-4b73-8370-ea937781c4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:32.190871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.190844    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4\" (UID: \"e005cb9b-d9a9-4b73-8370-ea937781c4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:32.191502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.191484    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4\" (UID: \"e005cb9b-d9a9-4b73-8370-ea937781c4fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:32.300086 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.300004    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:32.569563 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.569541    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"]
Sep 18 22:43:32.571351 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:43:32.571320    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode005cb9b_d9a9_4b73_8370_ea937781c4fa.slice/crio-dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965 WatchSource:0}: Error finding container dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965: Status 404 returned error can't find the container with id dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965
Sep 18 22:43:32.906377 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.906343    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4" event={"ID":"e005cb9b-d9a9-4b73-8370-ea937781c4fa","Type":"ContainerStarted","Data":"e8416e9e042fbe33e6396f84b649f5d9f9f7f7cc0eaa350925f3a3331e45d832"}
Sep 18 22:43:32.906696 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.906426    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4" event={"ID":"e005cb9b-d9a9-4b73-8370-ea937781c4fa","Type":"ContainerStarted","Data":"dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965"}
Sep 18 22:43:32.923210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:32.923166    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4" podStartSLOduration=1.9231515909999999 podStartE2EDuration="1.923151591s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:43:32.922120446 +0000 UTC m=+1605.103704991" watchObservedRunningTime="2026-09-18 22:43:32.923151591 +0000 UTC m=+1605.104736136"
Sep 18 22:43:33.913637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:33.913604    2551 generic.go:356] "Generic (PLEG): container finished" podID="e005cb9b-d9a9-4b73-8370-ea937781c4fa" containerID="e8416e9e042fbe33e6396f84b649f5d9f9f7f7cc0eaa350925f3a3331e45d832" exitCode=0
Sep 18 22:43:33.914011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:33.913660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4" event={"ID":"e005cb9b-d9a9-4b73-8370-ea937781c4fa","Type":"ContainerDied","Data":"e8416e9e042fbe33e6396f84b649f5d9f9f7f7cc0eaa350925f3a3331e45d832"}
Sep 18 22:43:35.050178 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.050153    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:35.115366 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.115338    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca\") pod \"e005cb9b-d9a9-4b73-8370-ea937781c4fa\" (UID: \"e005cb9b-d9a9-4b73-8370-ea937781c4fa\") "
Sep 18 22:43:35.115649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.115628    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e005cb9b-d9a9-4b73-8370-ea937781c4fa" (UID: "e005cb9b-d9a9-4b73-8370-ea937781c4fa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:35.216104 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.216012    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e005cb9b-d9a9-4b73-8370-ea937781c4fa-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:35.926221 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.926190    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"
Sep 18 22:43:35.926402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.926226    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4" event={"ID":"e005cb9b-d9a9-4b73-8370-ea937781c4fa","Type":"ContainerDied","Data":"dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965"}
Sep 18 22:43:35.926402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:35.926282    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd0a5a4a9882cf4aa8e8ef56dd55d9f7789bf295fac6c57c0aa510b49c4f2965"
Sep 18 22:43:36.899313 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.899289    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:43:36.931495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931419    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931464    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931507    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931532    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pjc8t\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931576    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931621    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931655    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931692 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931688    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931726    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert\") pod \"4fd057de-4441-4639-8445-db3979da3e40\" (UID: \"4fd057de-4441-4639-8445-db3979da3e40\") "
Sep 18 22:43:36.931971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931798    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.931971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.931921    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.932173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932001    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.932173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932124    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.932173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932142    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.932173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932155    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:36.932386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932258    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data" (OuterVolumeSpecName: "istio-data") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.932386 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.932283    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:36.933558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.933531    2551 generic.go:356] "Generic (PLEG): container finished" podID="4fd057de-4441-4639-8445-db3979da3e40" containerID="f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220" exitCode=0
Sep 18 22:43:36.933875 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.933600    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"
Sep 18 22:43:36.933875 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.933619    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerDied","Data":"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"}
Sep 18 22:43:36.933875 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.933694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42" event={"ID":"4fd057de-4441-4639-8445-db3979da3e40","Type":"ContainerDied","Data":"efc38bf5acf9f85d94ce7113bf2e6896e0d114be469264fc9bc3d714168d6e84"}
Sep 18 22:43:36.933875 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.933729    2551 scope.go:117] "RemoveContainer" containerID="f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"
Sep 18 22:43:36.934591 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.934553    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token" (OuterVolumeSpecName: "istio-token") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:36.934713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.934591    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:36.934713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.934693    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:43:36.934973 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.934905    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t" (OuterVolumeSpecName: "kube-api-access-pjc8t") pod "4fd057de-4441-4639-8445-db3979da3e40" (UID: "4fd057de-4441-4639-8445-db3979da3e40"). InnerVolumeSpecName "kube-api-access-pjc8t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:36.959115 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.959096    2551 scope.go:117] "RemoveContainer" containerID="5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"
Sep 18 22:43:36.968518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.968493    2551 scope.go:117] "RemoveContainer" containerID="f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"
Sep 18 22:43:36.968802 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:36.968786    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220\": container with ID starting with f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220 not found: ID does not exist" containerID="f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"
Sep 18 22:43:36.968863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.968809    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220"} err="failed to get container status \"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220\": rpc error: code = NotFound desc = could not find container \"f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220\": container with ID starting with f06b7991371bc93fd120161b9fdb63012567028247797de1284987a2e47ba220 not found: ID does not exist"
Sep 18 22:43:36.968863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.968825    2551 scope.go:117] "RemoveContainer" containerID="5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"
Sep 18 22:43:36.969143 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:36.969125    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f\": container with ID starting with 5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f not found: ID does not exist" containerID="5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"
Sep 18 22:43:36.969195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:36.969148    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f"} err="failed to get container status \"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f\": rpc error: code = NotFound desc = could not find container \"5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f\": container with ID starting with 5ef676c7b847781d5295fbfa90dfc8356eca1f831fcc08711ce925d64488fb9f not found: ID does not exist"
Sep 18 22:43:37.032823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032798    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/4fd057de-4441-4639-8445-db3979da3e40-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.032823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032823    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.032947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032832    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/4fd057de-4441-4639-8445-db3979da3e40-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.032947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032839    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.032947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032848    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pjc8t\" (UniqueName: \"kubernetes.io/projected/4fd057de-4441-4639-8445-db3979da3e40-kube-api-access-pjc8t\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.032947 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.032857    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/4fd057de-4441-4639-8445-db3979da3e40-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:37.257286 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.257258    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:43:37.263430 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:37.263399    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w2-8faaa1-openshift-default-5d964d9546-pcp42"]
Sep 18 22:43:38.290858 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.290816    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4fd057de-4441-4639-8445-db3979da3e40" path="/var/lib/kubelet/pods/4fd057de-4441-4639-8445-db3979da3e40/volumes"
Sep 18 22:43:38.690016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.689987    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"]
Sep 18 22:43:38.690409 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690396    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e005cb9b-d9a9-4b73-8370-ea937781c4fa" containerName="revoke-keys"
Sep 18 22:43:38.690457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690411    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e005cb9b-d9a9-4b73-8370-ea937781c4fa" containerName="revoke-keys"
Sep 18 22:43:38.690457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690426    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.690457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690431    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.690457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690437    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.690457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690442    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.690612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690495    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.690612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690503    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e005cb9b-d9a9-4b73-8370-ea937781c4fa" containerName="revoke-keys"
Sep 18 22:43:38.690612 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.690510    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4fd057de-4441-4639-8445-db3979da3e40" containerName="istio-proxy"
Sep 18 22:43:38.695067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.695046    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:38.698244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.698221    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:43:38.707188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.707170    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"]
Sep 18 22:43:38.752907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.752882    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml\" (UID: \"69a4be58-602a-46ba-a9d2-475d5ce16588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:38.853406 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.853382    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml\" (UID: \"69a4be58-602a-46ba-a9d2-475d5ce16588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:38.853962 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:38.853946    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml\" (UID: \"69a4be58-602a-46ba-a9d2-475d5ce16588\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:39.007677 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:39.007614    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:39.295183 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:39.295158    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"]
Sep 18 22:43:39.297572 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:43:39.297546    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod69a4be58_602a_46ba_a9d2_475d5ce16588.slice/crio-a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16 WatchSource:0}: Error finding container a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16: Status 404 returned error can't find the container with id a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16
Sep 18 22:43:39.953452 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:39.953423    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml" event={"ID":"69a4be58-602a-46ba-a9d2-475d5ce16588","Type":"ContainerStarted","Data":"9829f767daf4031d17e348fa0f3e1bcc0c6fe7a71e94e93457804cb9f9b0fef1"}
Sep 18 22:43:39.953592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:39.953475    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml" event={"ID":"69a4be58-602a-46ba-a9d2-475d5ce16588","Type":"ContainerStarted","Data":"a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16"}
Sep 18 22:43:39.973198 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:39.973158    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml" podStartSLOduration=1.973143613 podStartE2EDuration="1.973143613s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:43:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:43:39.970857239 +0000 UTC m=+1612.152441784" watchObservedRunningTime="2026-09-18 22:43:39.973143613 +0000 UTC m=+1612.154728159"
Sep 18 22:43:40.961133 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:40.961056    2551 generic.go:356] "Generic (PLEG): container finished" podID="69a4be58-602a-46ba-a9d2-475d5ce16588" containerID="9829f767daf4031d17e348fa0f3e1bcc0c6fe7a71e94e93457804cb9f9b0fef1" exitCode=0
Sep 18 22:43:40.961510 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:40.961142    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml" event={"ID":"69a4be58-602a-46ba-a9d2-475d5ce16588","Type":"ContainerDied","Data":"9829f767daf4031d17e348fa0f3e1bcc0c6fe7a71e94e93457804cb9f9b0fef1"}
Sep 18 22:43:42.101933 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.101909    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:42.179138 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.179107    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca\") pod \"69a4be58-602a-46ba-a9d2-475d5ce16588\" (UID: \"69a4be58-602a-46ba-a9d2-475d5ce16588\") "
Sep 18 22:43:42.179482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.179459    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "69a4be58-602a-46ba-a9d2-475d5ce16588" (UID: "69a4be58-602a-46ba-a9d2-475d5ce16588"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:42.280216 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.280147    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/69a4be58-602a-46ba-a9d2-475d5ce16588-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:42.973991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.973954    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml" event={"ID":"69a4be58-602a-46ba-a9d2-475d5ce16588","Type":"ContainerDied","Data":"a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16"}
Sep 18 22:43:42.973991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.973990    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"
Sep 18 22:43:42.974310 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:42.974067    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a6c71f5b378b246da930b3a727584c32e42772e314c1b4a6f4f6c9e57a3d8a16"
Sep 18 22:43:43.447686 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.447651    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:43:43.448123 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.447846    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" podUID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" containerName="payload-processing" containerID="cri-o://5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631" gracePeriod=30
Sep 18 22:43:43.662551 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.662477    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"]
Sep 18 22:43:43.669041 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.666490    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:43:43.673560 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.673492    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-20531f-f6fc4"]
Sep 18 22:43:43.703812 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.703759    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:43:43.793967 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.793941    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kr5h\" (UniqueName: \"kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h\") pod \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") "
Sep 18 22:43:43.794150 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.793998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume\") pod \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\" (UID: \"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0\") "
Sep 18 22:43:43.794400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.794379    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" (UID: "aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:43.796107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.796084    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h" (OuterVolumeSpecName: "kube-api-access-7kr5h") pod "aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" (UID: "aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0"). InnerVolumeSpecName "kube-api-access-7kr5h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:43.895741 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.895706    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kr5h\" (UniqueName: \"kubernetes.io/projected/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-kube-api-access-7kr5h\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:43.895741 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.895738    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:43.980751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.980670    2551 generic.go:356] "Generic (PLEG): container finished" podID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" containerID="5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631" exitCode=0
Sep 18 22:43:43.980751 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.980720    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"
Sep 18 22:43:43.980949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.980768    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" event={"ID":"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0","Type":"ContainerDied","Data":"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"}
Sep 18 22:43:43.980949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.980844    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr" event={"ID":"aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0","Type":"ContainerDied","Data":"8b36ba31e162b4a85bb5b30d0d7453ce1d1e48c553a9e5a6e2332cf16572dd1d"}
Sep 18 22:43:43.980949 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.980868    2551 scope.go:117] "RemoveContainer" containerID="5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"
Sep 18 22:43:43.990004 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.989986    2551 scope.go:117] "RemoveContainer" containerID="5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"
Sep 18 22:43:43.990318 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:43.990300    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631\": container with ID starting with 5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631 not found: ID does not exist" containerID="5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"
Sep 18 22:43:43.990372 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:43.990324    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631"} err="failed to get container status \"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631\": rpc error: code = NotFound desc = could not find container \"5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631\": container with ID starting with 5d146f04376c6da12027d484c1c7fb9ea482f80b0a6c62acb84459dd4d8f8631 not found: ID does not exist"
Sep 18 22:43:44.002955 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:44.002936    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:43:44.004732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:44.004713    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-20531f-6d974c5665-ldnmr"]
Sep 18 22:43:44.090165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:44.090142    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:43:44.291153 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:44.291075    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" path="/var/lib/kubelet/pods/aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0/volumes"
Sep 18 22:43:44.291494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:44.291475    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e005cb9b-d9a9-4b73-8370-ea937781c4fa" path="/var/lib/kubelet/pods/e005cb9b-d9a9-4b73-8370-ea937781c4fa/volumes"
Sep 18 22:43:50.523025 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:50.522993    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:43:50.523506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:50.523205    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy" containerID="cri-o://52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66" gracePeriod=30
Sep 18 22:43:54.380058 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.380005    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:43:54.380538 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.380258    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" podUID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" containerName="maas-api" containerID="cri-o://9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8" gracePeriod=30
Sep 18 22:43:54.412833 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.412800    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"]
Sep 18 22:43:54.415959 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.415936    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-f9cf33-29829510-jg7xq"]
Sep 18 22:43:54.703972 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.703943    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"]
Sep 18 22:43:54.710377 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.710346    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-f9cf33-ck5ml"]
Sep 18 22:43:54.830619 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.830600    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:43:54.989707 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.989626    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs\") pod \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") "
Sep 18 22:43:54.989707 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.989675    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca\") pod \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") "
Sep 18 22:43:54.989907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.989718    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") pod \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") "
Sep 18 22:43:54.989907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.989807    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pzqb5\" (UniqueName: \"kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5\") pod \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\" (UID: \"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b\") "
Sep 18 22:43:54.990079 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.990044    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca" (OuterVolumeSpecName: "openshift-service-ca") pod "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" (UID: "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:54.990175 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.990154    2551 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-openshift-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:54.991940 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.991916    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls" (OuterVolumeSpecName: "maas-api-tls") pod "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" (UID: "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:54.992042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.991934    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" (UID: "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:54.992042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:54.991945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5" (OuterVolumeSpecName: "kube-api-access-pzqb5") pod "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" (UID: "7ad8cf37-41e1-4b82-b415-ed0b6c49e54b"). InnerVolumeSpecName "kube-api-access-pzqb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:55.044921 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.044894    2551 generic.go:356] "Generic (PLEG): container finished" podID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" containerID="9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8" exitCode=0
Sep 18 22:43:55.045005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.044940    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"
Sep 18 22:43:55.045005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.044945    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" event={"ID":"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b","Type":"ContainerDied","Data":"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"}
Sep 18 22:43:55.045005 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.044987    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv" event={"ID":"7ad8cf37-41e1-4b82-b415-ed0b6c49e54b","Type":"ContainerDied","Data":"56cabe246fe0ad25c51a04b763fe51d27a38f9c72918b05fa31fb071ebced12f"}
Sep 18 22:43:55.045138 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.045009    2551 scope.go:117] "RemoveContainer" containerID="9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"
Sep 18 22:43:55.054277 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.054257    2551 scope.go:117] "RemoveContainer" containerID="9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"
Sep 18 22:43:55.054550 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:55.054529    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8\": container with ID starting with 9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8 not found: ID does not exist" containerID="9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"
Sep 18 22:43:55.054601 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.054558    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8"} err="failed to get container status \"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8\": rpc error: code = NotFound desc = could not find container \"9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8\": container with ID starting with 9a1f41227993b1c41e458434c64b758bf72f144a5a03ccdfdad3f26657b2aca8 not found: ID does not exist"
Sep 18 22:43:55.068072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.068048    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:43:55.074144 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.074125    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w4-f9cf33-75fdcd7475-mqbpv"]
Sep 18 22:43:55.091516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.091491    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-metrics-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.091516 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.091513    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-maas-api-tls\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.091629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.091527    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzqb5\" (UniqueName: \"kubernetes.io/projected/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b-kube-api-access-pzqb5\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.775212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.775192    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:43:55.901022 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.900998    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901091    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901127    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901199 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901171    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901237    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901295    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-2k844\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901334    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:55.901420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901376    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data" (OuterVolumeSpecName: "istio-data") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:55.901420 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901392    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901473    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token\") pod \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\" (UID: \"a4abbdf1-c220-4d58-8dcc-c7318c62130b\") "
Sep 18 22:43:55.901659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901494    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:55.901659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901634    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:55.901807 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901716    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:43:55.901866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901804    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.901866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901822    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.901866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901837    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.901866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901850    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.901866 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.901862    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:55.903375 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.903352    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:43:55.903503 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.903483    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token" (OuterVolumeSpecName: "istio-token") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:55.903792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.903770    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:55.903851 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:55.903788    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844" (OuterVolumeSpecName: "kube-api-access-2k844") pod "a4abbdf1-c220-4d58-8dcc-c7318c62130b" (UID: "a4abbdf1-c220-4d58-8dcc-c7318c62130b"). InnerVolumeSpecName "kube-api-access-2k844". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:56.002659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.002634    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:56.002659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.002654    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:56.002791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.002665    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2k844\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-kube-api-access-2k844\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:56.002791 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.002674    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4abbdf1-c220-4d58-8dcc-c7318c62130b-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:56.052508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.052486    2551 generic.go:356] "Generic (PLEG): container finished" podID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerID="52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66" exitCode=0
Sep 18 22:43:56.052583 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.052518    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerDied","Data":"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"}
Sep 18 22:43:56.052583 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.052571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m" event={"ID":"a4abbdf1-c220-4d58-8dcc-c7318c62130b","Type":"ContainerDied","Data":"3a30815a4c786da6feb8cf7e8aff8ac1909ff2f020327ddf63df6bc4bf77760d"}
Sep 18 22:43:56.052669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.052535    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"
Sep 18 22:43:56.052669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.052593    2551 scope.go:117] "RemoveContainer" containerID="52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"
Sep 18 22:43:56.061911 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.061896    2551 scope.go:117] "RemoveContainer" containerID="fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"
Sep 18 22:43:56.069756 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.069736    2551 scope.go:117] "RemoveContainer" containerID="52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"
Sep 18 22:43:56.070011 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:56.069995    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66\": container with ID starting with 52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66 not found: ID does not exist" containerID="52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"
Sep 18 22:43:56.070116 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.070022    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66"} err="failed to get container status \"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66\": rpc error: code = NotFound desc = could not find container \"52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66\": container with ID starting with 52d9822e17f01041e621cbc2c58b856b8684c9033ec9f044603491cfdd3b5a66 not found: ID does not exist"
Sep 18 22:43:56.070116 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.070060    2551 scope.go:117] "RemoveContainer" containerID="fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"
Sep 18 22:43:56.070315 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:56.070297    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe\": container with ID starting with fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe not found: ID does not exist" containerID="fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"
Sep 18 22:43:56.070356 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.070322    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe"} err="failed to get container status \"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe\": rpc error: code = NotFound desc = could not find container \"fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe\": container with ID starting with fb4c669b2ecdc4adc96ea09cab4d9693c75666c09150aa28e665e2227ae34cbe not found: ID does not exist"
Sep 18 22:43:56.074894 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.074873    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:43:56.079109 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.079089    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w2-20531f-openshift-default-5d6fd9c6d6-xjg7m"]
Sep 18 22:43:56.291483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.291424    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="69a4be58-602a-46ba-a9d2-475d5ce16588" path="/var/lib/kubelet/pods/69a4be58-602a-46ba-a9d2-475d5ce16588/volumes"
Sep 18 22:43:56.291792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.291780    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" path="/var/lib/kubelet/pods/7ad8cf37-41e1-4b82-b415-ed0b6c49e54b/volumes"
Sep 18 22:43:56.292018 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.292006    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="89b4f896-ecf4-477e-ab9f-2cd8e66dced5" path="/var/lib/kubelet/pods/89b4f896-ecf4-477e-ab9f-2cd8e66dced5/volumes"
Sep 18 22:43:56.292360 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:56.292348    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" path="/var/lib/kubelet/pods/a4abbdf1-c220-4d58-8dcc-c7318c62130b/volumes"
Sep 18 22:43:58.690533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:58.690495    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" podUID="64349f74-2f8c-4f35-962a-2c27f88dd31a" containerName="main" containerID="cri-o://a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8" gracePeriod=45
Sep 18 22:43:58.934985 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:58.934961    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:43:59.028248 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028165    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028249    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028266    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028287    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028305    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028411 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028337    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qtdl9\" (UniqueName: \"kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9\") pod \"64349f74-2f8c-4f35-962a-2c27f88dd31a\" (UID: \"64349f74-2f8c-4f35-962a-2c27f88dd31a\") "
Sep 18 22:43:59.028684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028452    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home" (OuterVolumeSpecName: "home") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:59.028684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028551    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache" (OuterVolumeSpecName: "model-cache") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:59.028684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028563    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:59.028827 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028738    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:59.028827 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028755    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:59.028827 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.028771    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:59.030558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.030537    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm" (OuterVolumeSpecName: "dshm") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:43:59.030687 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.030670    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:43:59.030870 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.030855    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9" (OuterVolumeSpecName: "kube-api-access-qtdl9") pod "64349f74-2f8c-4f35-962a-2c27f88dd31a" (UID: "64349f74-2f8c-4f35-962a-2c27f88dd31a"). InnerVolumeSpecName "kube-api-access-qtdl9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:43:59.076500 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.076466    2551 generic.go:356] "Generic (PLEG): container finished" podID="64349f74-2f8c-4f35-962a-2c27f88dd31a" containerID="a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8" exitCode=0
Sep 18 22:43:59.076651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.076523    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"
Sep 18 22:43:59.076651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.076544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" event={"ID":"64349f74-2f8c-4f35-962a-2c27f88dd31a","Type":"ContainerDied","Data":"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"}
Sep 18 22:43:59.076651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.076586    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl" event={"ID":"64349f74-2f8c-4f35-962a-2c27f88dd31a","Type":"ContainerDied","Data":"2c3f7db33540db4a5e8e7db6824aa56d3f3f3135ed173bac06ad6360b62e13fb"}
Sep 18 22:43:59.076651 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.076615    2551 scope.go:117] "RemoveContainer" containerID="a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"
Sep 18 22:43:59.085324 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.085295    2551 scope.go:117] "RemoveContainer" containerID="a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"
Sep 18 22:43:59.085690 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:43:59.085670    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8\": container with ID starting with a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8 not found: ID does not exist" containerID="a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"
Sep 18 22:43:59.085772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.085695    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8"} err="failed to get container status \"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8\": rpc error: code = NotFound desc = could not find container \"a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8\": container with ID starting with a92481383b42d5fdde8aa3cbb3f2aa948599f1a3bd3097989bede9767cfd72b8 not found: ID does not exist"
Sep 18 22:43:59.096817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.096799    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:43:59.100022 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.100005    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w4-f9cf33/simulator-w4-f9cf33-kserve-69b7dccf-v6hrl"]
Sep 18 22:43:59.129898 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.129877    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/64349f74-2f8c-4f35-962a-2c27f88dd31a-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:59.129898 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.129897    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/64349f74-2f8c-4f35-962a-2c27f88dd31a-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:43:59.130016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:43:59.129907    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qtdl9\" (UniqueName: \"kubernetes.io/projected/64349f74-2f8c-4f35-962a-2c27f88dd31a-kube-api-access-qtdl9\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:44:00.289968 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:00.289937    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="64349f74-2f8c-4f35-962a-2c27f88dd31a" path="/var/lib/kubelet/pods/64349f74-2f8c-4f35-962a-2c27f88dd31a/volumes"
Sep 18 22:44:07.097142 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:07.097106    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:44:10.021907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.021875    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-a076c6"]
Sep 18 22:44:10.022306 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022292    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" containerName="payload-processing"
Sep 18 22:44:10.022306 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022306    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" containerName="payload-processing"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022314    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022319    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022332    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="64349f74-2f8c-4f35-962a-2c27f88dd31a" containerName="main"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022340    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="64349f74-2f8c-4f35-962a-2c27f88dd31a" containerName="main"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022348    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022353    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022368    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" containerName="maas-api"
Sep 18 22:44:10.022376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022373    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" containerName="maas-api"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022380    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="69a4be58-602a-46ba-a9d2-475d5ce16588" containerName="revoke-keys"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022385    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="69a4be58-602a-46ba-a9d2-475d5ce16588" containerName="revoke-keys"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022451    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="69a4be58-602a-46ba-a9d2-475d5ce16588" containerName="revoke-keys"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022459    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7ad8cf37-41e1-4b82-b415-ed0b6c49e54b" containerName="maas-api"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022465    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="64349f74-2f8c-4f35-962a-2c27f88dd31a" containerName="main"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022471    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022476    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a4abbdf1-c220-4d58-8dcc-c7318c62130b" containerName="istio-proxy"
Sep 18 22:44:10.022608 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.022485    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aeb9eb8a-4e2f-411c-a04e-0d46dc5ac2f0" containerName="payload-processing"
Sep 18 22:44:10.026881 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.026862    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:10.029633 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.029611    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lgx4p\""
Sep 18 22:44:10.035141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.035119    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-a076c6"]
Sep 18 22:44:10.215627 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.215599    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9swtr\" (UniqueName: \"kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr\") pod \"test-curl-7544-a076c6\" (UID: \"d9e9c996-2d71-4a69-92b5-9a5a4033e250\") " pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:10.316650 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.316578    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9swtr\" (UniqueName: \"kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr\") pod \"test-curl-7544-a076c6\" (UID: \"d9e9c996-2d71-4a69-92b5-9a5a4033e250\") " pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:10.325448 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.325426    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9swtr\" (UniqueName: \"kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr\") pod \"test-curl-7544-a076c6\" (UID: \"d9e9c996-2d71-4a69-92b5-9a5a4033e250\") " pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:10.337285 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.337262    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:10.608511 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:10.608413    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-a076c6"]
Sep 18 22:44:10.611377 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:44:10.611356    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podd9e9c996_2d71_4a69_92b5_9a5a4033e250.slice/crio-96d771cd0c2796f3e4007ca89c01338dce52e4325b03a6894806f7e168729fdc WatchSource:0}: Error finding container 96d771cd0c2796f3e4007ca89c01338dce52e4325b03a6894806f7e168729fdc: Status 404 returned error can't find the container with id 96d771cd0c2796f3e4007ca89c01338dce52e4325b03a6894806f7e168729fdc
Sep 18 22:44:11.144105 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:11.144067    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-a076c6" event={"ID":"d9e9c996-2d71-4a69-92b5-9a5a4033e250","Type":"ContainerStarted","Data":"08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f"}
Sep 18 22:44:11.144105 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:11.144119    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-a076c6" event={"ID":"d9e9c996-2d71-4a69-92b5-9a5a4033e250","Type":"ContainerStarted","Data":"96d771cd0c2796f3e4007ca89c01338dce52e4325b03a6894806f7e168729fdc"}
Sep 18 22:44:11.162037 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:11.161993    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-a076c6" podStartSLOduration=1.161978398 podStartE2EDuration="1.161978398s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:44:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:44:11.160169797 +0000 UTC m=+1643.341754357" watchObservedRunningTime="2026-09-18 22:44:11.161978398 +0000 UTC m=+1643.343562945"
Sep 18 22:44:11.760425 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:11.760394    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-a076c6"]
Sep 18 22:44:11.765562 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:11.765538    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-a076c6"]
Sep 18 22:44:12.150011 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:44:12.149985    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-a076c6" container="test-curl-7544-a076c6"
Sep 18 22:44:13.154966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:13.154922    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-a076c6" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" containerName="test-curl-7544-a076c6" containerID="cri-o://08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f" gracePeriod=2
Sep 18 22:44:15.341993 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:15.341969    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:15.354121 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:15.354102    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9swtr\" (UniqueName: \"kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr\") pod \"d9e9c996-2d71-4a69-92b5-9a5a4033e250\" (UID: \"d9e9c996-2d71-4a69-92b5-9a5a4033e250\") "
Sep 18 22:44:15.355913 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:15.355888    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr" (OuterVolumeSpecName: "kube-api-access-9swtr") pod "d9e9c996-2d71-4a69-92b5-9a5a4033e250" (UID: "d9e9c996-2d71-4a69-92b5-9a5a4033e250"). InnerVolumeSpecName "kube-api-access-9swtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:44:15.455023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:15.454973    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9swtr\" (UniqueName: \"kubernetes.io/projected/d9e9c996-2d71-4a69-92b5-9a5a4033e250-kube-api-access-9swtr\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:44:16.173100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.173058    2551 generic.go:356] "Generic (PLEG): container finished" podID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" containerID="08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f" exitCode=137
Sep 18 22:44:16.173261 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.173157    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:16.173261 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.173202    2551 scope.go:117] "RemoveContainer" containerID="08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f"
Sep 18 22:44:16.182374 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.182355    2551 scope.go:117] "RemoveContainer" containerID="08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f"
Sep 18 22:44:16.182633 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:44:16.182615    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f\": container with ID starting with 08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f not found: ID does not exist" containerID="08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f"
Sep 18 22:44:16.182684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.182641    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f"} err="failed to get container status \"08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f\": rpc error: code = NotFound desc = could not find container \"08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f\": container with ID starting with 08b23b83d8a818916caee71cb421acfde7ee74e864e7ae7d337eb4f5cd96f96f not found: ID does not exist"
Sep 18 22:44:16.183755 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:44:16.183730    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"test-curl-7544-a076c6\" is forbidden: User \"system:node:ip-10-0-129-37.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"opendatahub\": no relationship found between node 'ip-10-0-129-37.ec2.internal' and this object" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" pod="opendatahub/test-curl-7544-a076c6"
Sep 18 22:44:16.290758 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.290731    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" path="/var/lib/kubelet/pods/d9e9c996-2d71-4a69-92b5-9a5a4033e250/volumes"
Sep 18 22:44:16.899969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:16.899940    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:44:40.007392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:40.007351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:44:48.623643 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:44:48.623612    2551 scope.go:117] "RemoveContainer" containerID="df3ab15bb5d7cc1bf7c14b1ed09e964a939cbca5d9c441740836fd8a46ee598c"
Sep 18 22:45:00.139322 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.139284    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"]
Sep 18 22:45:00.139700 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.139690    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" containerName="test-curl-7544-a076c6"
Sep 18 22:45:00.139743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.139702    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" containerName="test-curl-7544-a076c6"
Sep 18 22:45:00.139778 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.139773    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d9e9c996-2d71-4a69-92b5-9a5a4033e250" containerName="test-curl-7544-a076c6"
Sep 18 22:45:00.142975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.142950    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"]
Sep 18 22:45:00.142975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.142965    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"
Sep 18 22:45:00.145427 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.145403    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:45:00.146239 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.146220    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"
Sep 18 22:45:00.151611 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.151576    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"]
Sep 18 22:45:00.154652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.154632    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"
Sep 18 22:45:00.161966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.161938    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"]
Sep 18 22:45:00.164253 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.164235    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"
Sep 18 22:45:00.440915 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.440892    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"]
Sep 18 22:45:00.443461 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:45:00.443427    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e023a35_19da_4261_b88e_7202e6d64d93.slice/crio-04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d WatchSource:0}: Error finding container 04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d: Status 404 returned error can't find the container with id 04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d
Sep 18 22:45:00.446334 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:00.446254    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"]
Sep 18 22:45:00.448165 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:45:00.448141    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddff96a01_2731_426a_b32c_1e222614194d.slice/crio-ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b WatchSource:0}: Error finding container ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b: Status 404 returned error can't find the container with id ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b
Sep 18 22:45:01.422243 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.422214    2551 generic.go:356] "Generic (PLEG): container finished" podID="5e023a35-19da-4261-b88e-7202e6d64d93" containerID="cb7aea5a800e5d07c74e781215e071fc8ba9f40092d0c076ecb8c3ffc052ba95" exitCode=0
Sep 18 22:45:01.422621 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.422275    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5" event={"ID":"5e023a35-19da-4261-b88e-7202e6d64d93","Type":"ContainerDied","Data":"cb7aea5a800e5d07c74e781215e071fc8ba9f40092d0c076ecb8c3ffc052ba95"}
Sep 18 22:45:01.422621 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.422334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5" event={"ID":"5e023a35-19da-4261-b88e-7202e6d64d93","Type":"ContainerStarted","Data":"04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d"}
Sep 18 22:45:01.423925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.423904    2551 generic.go:356] "Generic (PLEG): container finished" podID="dff96a01-2731-426a-b32c-1e222614194d" containerID="f327fc03c055d6f7ab89e20c0b7bd31550aeba3f75569b236a84db49db04fdbf" exitCode=0
Sep 18 22:45:01.424022 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.423956    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565" event={"ID":"dff96a01-2731-426a-b32c-1e222614194d","Type":"ContainerDied","Data":"f327fc03c055d6f7ab89e20c0b7bd31550aeba3f75569b236a84db49db04fdbf"}
Sep 18 22:45:01.424022 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:01.423987    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565" event={"ID":"dff96a01-2731-426a-b32c-1e222614194d","Type":"ContainerStarted","Data":"ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b"}
Sep 18 22:45:02.578043 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:02.578008    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"
Sep 18 22:45:02.580975 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:02.580956    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"
Sep 18 22:45:03.145702 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.145669    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:45:03.146070 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146057    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="dff96a01-2731-426a-b32c-1e222614194d" containerName="cleanup"
Sep 18 22:45:03.146125 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146071    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="dff96a01-2731-426a-b32c-1e222614194d" containerName="cleanup"
Sep 18 22:45:03.146125 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146080    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5e023a35-19da-4261-b88e-7202e6d64d93" containerName="cleanup"
Sep 18 22:45:03.146125 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146085    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5e023a35-19da-4261-b88e-7202e6d64d93" containerName="cleanup"
Sep 18 22:45:03.146225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146156    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5e023a35-19da-4261-b88e-7202e6d64d93" containerName="cleanup"
Sep 18 22:45:03.146225 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.146165    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="dff96a01-2731-426a-b32c-1e222614194d" containerName="cleanup"
Sep 18 22:45:03.149257 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.149238    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.151908 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.151888    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"kube-root-ca.crt\""
Sep 18 22:45:03.153175 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.153157    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"default-dockercfg-vphth\""
Sep 18 22:45:03.153272 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.153198    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"openshift-service-ca.crt\""
Sep 18 22:45:03.153339 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.153289    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"sub-test-model-a91c21-kserve-self-signed-certs\""
Sep 18 22:45:03.158124 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.158104    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:45:03.232552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232524    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.232659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232568    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.232659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232583    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.232659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232640    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.232834 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232706    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tb7s6\" (UniqueName: \"kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.232834 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.232794    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333300 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333276    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333426 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333308    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tb7s6\" (UniqueName: \"kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333426 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333349    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333544 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333412    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333544 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333544 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333877 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333854    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333959 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.333959 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.333929    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.335653 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.335630    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.335737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.335689    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.342115 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.342094    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tb7s6\" (UniqueName: \"kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6\") pod \"sub-test-model-a91c21-kserve-66888dcb97-wv58v\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.437273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.437219    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"
Sep 18 22:45:03.437273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.437258    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565" event={"ID":"dff96a01-2731-426a-b32c-1e222614194d","Type":"ContainerDied","Data":"ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b"}
Sep 18 22:45:03.437399 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.437306    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac2ecf17eeb11edd06620f4a11875438d8e1954e34dfdaad635c0b28fc6f6e6b"
Sep 18 22:45:03.438604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.438589    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5"
Sep 18 22:45:03.438604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.438597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29829525-6gjr5" event={"ID":"5e023a35-19da-4261-b88e-7202e6d64d93","Type":"ContainerDied","Data":"04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d"}
Sep 18 22:45:03.438724 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.438627    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="04c8099be20485c4d619a0e527f88973f281e728cb9771625da05833e1bd3d4d"
Sep 18 22:45:03.459659 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.459641    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:03.732817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:03.732731    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:45:03.735564 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:45:03.735532    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod84c3cf2a_6e97_4e38_8be9_b61ccb52ae83.slice/crio-37c229943996b8c199ad0b6aa7935dd1e85ad89bf0ee73c419865483096269f2 WatchSource:0}: Error finding container 37c229943996b8c199ad0b6aa7935dd1e85ad89bf0ee73c419865483096269f2: Status 404 returned error can't find the container with id 37c229943996b8c199ad0b6aa7935dd1e85ad89bf0ee73c419865483096269f2
Sep 18 22:45:04.446966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:04.446932    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" event={"ID":"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83","Type":"ContainerStarted","Data":"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"}
Sep 18 22:45:04.447158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:04.446975    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" event={"ID":"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83","Type":"ContainerStarted","Data":"37c229943996b8c199ad0b6aa7935dd1e85ad89bf0ee73c419865483096269f2"}
Sep 18 22:45:04.468069 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:04.468012    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" podStartSLOduration=1.467997308 podStartE2EDuration="1.467997308s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:04.466446374 +0000 UTC m=+1696.648030924" watchObservedRunningTime="2026-09-18 22:45:04.467997308 +0000 UTC m=+1696.649581853"
Sep 18 22:45:06.803769 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:06.803740    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:45:13.460782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:13.460700    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:13.460782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:13.460750    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:13.473630 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:13.473606    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:13.506907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:13.506886    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:27.896756 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:27.896722    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:45:36.211991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.211949    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-4db58c"]
Sep 18 22:45:36.217207 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.217183    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:36.219801 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.219775    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"default-dockercfg-lgx4p\""
Sep 18 22:45:36.221899 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.221872    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-4db58c"]
Sep 18 22:45:36.279623 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.279597    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7qgv\" (UniqueName: \"kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv\") pod \"test-curl-7544-4db58c\" (UID: \"6e8cf512-b852-4fdf-bc0e-48d98df945bf\") " pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:36.380418 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.380395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7qgv\" (UniqueName: \"kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv\") pod \"test-curl-7544-4db58c\" (UID: \"6e8cf512-b852-4fdf-bc0e-48d98df945bf\") " pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:36.388823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.388802    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7qgv\" (UniqueName: \"kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv\") pod \"test-curl-7544-4db58c\" (UID: \"6e8cf512-b852-4fdf-bc0e-48d98df945bf\") " pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:36.528603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.528548    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:36.801173 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:36.801149    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-4db58c"]
Sep 18 22:45:36.802497 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:45:36.802471    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6e8cf512_b852_4fdf_bc0e_48d98df945bf.slice/crio-cb3d2bd3eb2b31c0a5587645059d341697c2e1e1e673171a80dfe0e6837211e8 WatchSource:0}: Error finding container cb3d2bd3eb2b31c0a5587645059d341697c2e1e1e673171a80dfe0e6837211e8: Status 404 returned error can't find the container with id cb3d2bd3eb2b31c0a5587645059d341697c2e1e1e673171a80dfe0e6837211e8
Sep 18 22:45:37.632606 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:37.632566    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-4db58c" event={"ID":"6e8cf512-b852-4fdf-bc0e-48d98df945bf","Type":"ContainerStarted","Data":"4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6"}
Sep 18 22:45:37.632999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:37.632627    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-4db58c" event={"ID":"6e8cf512-b852-4fdf-bc0e-48d98df945bf","Type":"ContainerStarted","Data":"cb3d2bd3eb2b31c0a5587645059d341697c2e1e1e673171a80dfe0e6837211e8"}
Sep 18 22:45:37.648466 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:37.648423    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-4db58c" podStartSLOduration=1.648409004 podStartE2EDuration="1.648409004s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:37.646740991 +0000 UTC m=+1729.828325540" watchObservedRunningTime="2026-09-18 22:45:37.648409004 +0000 UTC m=+1729.829993549"
Sep 18 22:45:38.275371 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.275343    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-4db58c"]
Sep 18 22:45:38.278457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.278428    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-4db58c"]
Sep 18 22:45:38.393297 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.393274    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/test-curl-7544-314c5b"]
Sep 18 22:45:38.393669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.393655    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" containerName="test-curl-7544-4db58c"
Sep 18 22:45:38.393669 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.393669    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" containerName="test-curl-7544-4db58c"
Sep 18 22:45:38.393762 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.393719    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" containerName="test-curl-7544-4db58c"
Sep 18 22:45:38.396817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.396799    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:38.403741 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.403493    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-314c5b"]
Sep 18 22:45:38.500262 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.500239    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hvtng\" (UniqueName: \"kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng\") pod \"test-curl-7544-314c5b\" (UID: \"b964731f-aae8-426a-856c-0605e1ab7fe2\") " pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:38.600713 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.600687    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hvtng\" (UniqueName: \"kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng\") pod \"test-curl-7544-314c5b\" (UID: \"b964731f-aae8-426a-856c-0605e1ab7fe2\") " pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:38.608720 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.608697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hvtng\" (UniqueName: \"kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng\") pod \"test-curl-7544-314c5b\" (UID: \"b964731f-aae8-426a-856c-0605e1ab7fe2\") " pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:38.638643 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:45:38.638612    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-4db58c" container="test-curl-7544-4db58c"
Sep 18 22:45:38.706674 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.706616    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:38.977446 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:38.977422    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/test-curl-7544-314c5b"]
Sep 18 22:45:38.979516 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:45:38.979489    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb964731f_aae8_426a_856c_0605e1ab7fe2.slice/crio-d0466bbd18bc1ab9189864f551dcfbccb8c9c8a7044adddbbb28ae2d7ca70b02 WatchSource:0}: Error finding container d0466bbd18bc1ab9189864f551dcfbccb8c9c8a7044adddbbb28ae2d7ca70b02: Status 404 returned error can't find the container with id d0466bbd18bc1ab9189864f551dcfbccb8c9c8a7044adddbbb28ae2d7ca70b02
Sep 18 22:45:39.645695 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:39.645661    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-314c5b" event={"ID":"b964731f-aae8-426a-856c-0605e1ab7fe2","Type":"ContainerStarted","Data":"14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e"}
Sep 18 22:45:39.646148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:39.645707    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/test-curl-7544-314c5b" event={"ID":"b964731f-aae8-426a-856c-0605e1ab7fe2","Type":"ContainerStarted","Data":"d0466bbd18bc1ab9189864f551dcfbccb8c9c8a7044adddbbb28ae2d7ca70b02"}
Sep 18 22:45:39.646148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:39.645873    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-4db58c" podUID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" containerName="test-curl-7544-4db58c" containerID="cri-o://4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6" gracePeriod=2
Sep 18 22:45:39.661577 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:39.661534    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/test-curl-7544-314c5b" podStartSLOduration=1.661519689 podStartE2EDuration="1.661519689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:45:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:45:39.659744117 +0000 UTC m=+1731.841328663" watchObservedRunningTime="2026-09-18 22:45:39.661519689 +0000 UTC m=+1731.843104235"
Sep 18 22:45:40.260584 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:40.260555    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/test-curl-7544-314c5b"]
Sep 18 22:45:40.267731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:40.267707    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/test-curl-7544-314c5b"]
Sep 18 22:45:40.651339 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:45:40.651312    2551 kuberuntime_manager.go:649] "Missing actuated resource record" pod="opendatahub/test-curl-7544-314c5b" container="test-curl-7544-314c5b"
Sep 18 22:45:41.297722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:41.297685    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:45:41.657801 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:41.657769    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/test-curl-7544-314c5b" podUID="b964731f-aae8-426a-856c-0605e1ab7fe2" containerName="test-curl-7544-314c5b" containerID="cri-o://14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e" gracePeriod=2
Sep 18 22:45:41.808782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:41.808756    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:41.926546 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:41.926487    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-d7qgv\" (UniqueName: \"kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv\") pod \"6e8cf512-b852-4fdf-bc0e-48d98df945bf\" (UID: \"6e8cf512-b852-4fdf-bc0e-48d98df945bf\") "
Sep 18 22:45:41.928434 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:41.928400    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv" (OuterVolumeSpecName: "kube-api-access-d7qgv") pod "6e8cf512-b852-4fdf-bc0e-48d98df945bf" (UID: "6e8cf512-b852-4fdf-bc0e-48d98df945bf"). InnerVolumeSpecName "kube-api-access-d7qgv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:42.027508 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.027475    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d7qgv\" (UniqueName: \"kubernetes.io/projected/6e8cf512-b852-4fdf-bc0e-48d98df945bf-kube-api-access-d7qgv\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:42.290592 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.290521    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" path="/var/lib/kubelet/pods/6e8cf512-b852-4fdf-bc0e-48d98df945bf/volumes"
Sep 18 22:45:42.665632 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.665599    2551 generic.go:356] "Generic (PLEG): container finished" podID="6e8cf512-b852-4fdf-bc0e-48d98df945bf" containerID="4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6" exitCode=137
Sep 18 22:45:42.665988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.665654    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-4db58c"
Sep 18 22:45:42.665988 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.665696    2551 scope.go:117] "RemoveContainer" containerID="4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6"
Sep 18 22:45:42.677744 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.677728    2551 scope.go:117] "RemoveContainer" containerID="4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6"
Sep 18 22:45:42.678051 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:45:42.678008    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6\": container with ID starting with 4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6 not found: ID does not exist" containerID="4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6"
Sep 18 22:45:42.678128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:42.678048    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6"} err="failed to get container status \"4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6\": rpc error: code = NotFound desc = could not find container \"4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6\": container with ID starting with 4e04119908ae58bac034b70cbebc78d972e76a8f355b8a8665c70beaf439b6c6 not found: ID does not exist"
Sep 18 22:45:43.238646 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:43.238616    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:45:43.835326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:43.835300    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:43.944302 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:43.944245    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hvtng\" (UniqueName: \"kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng\") pod \"b964731f-aae8-426a-856c-0605e1ab7fe2\" (UID: \"b964731f-aae8-426a-856c-0605e1ab7fe2\") "
Sep 18 22:45:43.946191 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:43.946160    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng" (OuterVolumeSpecName: "kube-api-access-hvtng") pod "b964731f-aae8-426a-856c-0605e1ab7fe2" (UID: "b964731f-aae8-426a-856c-0605e1ab7fe2"). InnerVolumeSpecName "kube-api-access-hvtng". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:44.045771 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.045748    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hvtng\" (UniqueName: \"kubernetes.io/projected/b964731f-aae8-426a-856c-0605e1ab7fe2-kube-api-access-hvtng\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:44.291818 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.291738    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b964731f-aae8-426a-856c-0605e1ab7fe2" path="/var/lib/kubelet/pods/b964731f-aae8-426a-856c-0605e1ab7fe2/volumes"
Sep 18 22:45:44.679658 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.679624    2551 generic.go:356] "Generic (PLEG): container finished" podID="b964731f-aae8-426a-856c-0605e1ab7fe2" containerID="14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e" exitCode=137
Sep 18 22:45:44.679815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.679671    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/test-curl-7544-314c5b"
Sep 18 22:45:44.679815 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.679712    2551 scope.go:117] "RemoveContainer" containerID="14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e"
Sep 18 22:45:44.688312 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.688285    2551 scope.go:117] "RemoveContainer" containerID="14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e"
Sep 18 22:45:44.688589 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:45:44.688569    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e\": container with ID starting with 14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e not found: ID does not exist" containerID="14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e"
Sep 18 22:45:44.688668 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:44.688601    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e"} err="failed to get container status \"14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e\": rpc error: code = NotFound desc = could not find container \"14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e\": container with ID starting with 14a464aeaac0d1f06647ef0ef249cf7725ef886bbc67d038b337402fc43ab28e not found: ID does not exist"
Sep 18 22:45:48.662365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:48.662333    2551 scope.go:117] "RemoveContainer" containerID="330b45b0374f544e49a26ed2438d0cdf5a605fc112db24808f22379d1a0de472"
Sep 18 22:45:52.599426 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:52.599394    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:45:58.257622 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.257579    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" podUID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" containerName="main" containerID="cri-o://136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9" gracePeriod=45
Sep 18 22:45:58.508997 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.508943    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:58.658203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658175    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tb7s6\" (UniqueName: \"kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658214    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658232    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658312    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658367    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658398    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home\") pod \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\" (UID: \"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83\") "
Sep 18 22:45:58.658558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658464    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache" (OuterVolumeSpecName: "model-cache") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:58.658670 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658640    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:58.658722 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658667    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home" (OuterVolumeSpecName: "home") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:58.658775 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.658738    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.660426 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.660400    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm" (OuterVolumeSpecName: "dshm") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:45:58.660534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.660449    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:45:58.660534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.660490    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6" (OuterVolumeSpecName: "kube-api-access-tb7s6") pod "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" (UID: "84c3cf2a-6e97-4e38-8be9-b61ccb52ae83"). InnerVolumeSpecName "kube-api-access-tb7s6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:45:58.753340 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.753317    2551 generic.go:356] "Generic (PLEG): container finished" podID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" containerID="136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9" exitCode=0
Sep 18 22:45:58.753456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.753357    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"
Sep 18 22:45:58.753456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.753383    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" event={"ID":"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83","Type":"ContainerDied","Data":"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"}
Sep 18 22:45:58.753456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.753421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v" event={"ID":"84c3cf2a-6e97-4e38-8be9-b61ccb52ae83","Type":"ContainerDied","Data":"37c229943996b8c199ad0b6aa7935dd1e85ad89bf0ee73c419865483096269f2"}
Sep 18 22:45:58.753456 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.753441    2551 scope.go:117] "RemoveContainer" containerID="136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"
Sep 18 22:45:58.759148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.759094    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.759148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.759114    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.759148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.759123    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.759148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.759132    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tb7s6\" (UniqueName: \"kubernetes.io/projected/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-kube-api-access-tb7s6\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.759148 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.759140    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:45:58.762867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.762844    2551 scope.go:117] "RemoveContainer" containerID="136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"
Sep 18 22:45:58.763155 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:45:58.763135    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9\": container with ID starting with 136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9 not found: ID does not exist" containerID="136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"
Sep 18 22:45:58.763219 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.763165    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9"} err="failed to get container status \"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9\": rpc error: code = NotFound desc = could not find container \"136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9\": container with ID starting with 136ee336417a9d52040f926cc20894224613febf7f71b4dd82bd325095a1bfd9 not found: ID does not exist"
Sep 18 22:45:58.777474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.777449    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:45:58.781886 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:45:58.781865    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/sub-test-model-a91c21-kserve-66888dcb97-wv58v"]
Sep 18 22:46:00.290403 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:00.290361    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" path="/var/lib/kubelet/pods/84c3cf2a-6e97-4e38-8be9-b61ccb52ae83/volumes"
Sep 18 22:46:13.024724 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.024691    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:46:13.025102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025056    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b964731f-aae8-426a-856c-0605e1ab7fe2" containerName="test-curl-7544-314c5b"
Sep 18 22:46:13.025102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025068    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b964731f-aae8-426a-856c-0605e1ab7fe2" containerName="test-curl-7544-314c5b"
Sep 18 22:46:13.025102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025093    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" containerName="main"
Sep 18 22:46:13.025102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025099    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" containerName="main"
Sep 18 22:46:13.025247 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025154    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b964731f-aae8-426a-856c-0605e1ab7fe2" containerName="test-curl-7544-314c5b"
Sep 18 22:46:13.025247 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.025164    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="84c3cf2a-6e97-4e38-8be9-b61ccb52ae83" containerName="main"
Sep 18 22:46:13.029561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.029539    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.033441 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.033418    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"kube-root-ca.crt\""
Sep 18 22:46:13.034445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.034425    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"default-dockercfg-vphth\""
Sep 18 22:46:13.034558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.034521    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"openshift-service-ca.crt\""
Sep 18 22:46:13.034625 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.034568    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w3-a91c21\"/\"rate-test-model-a91c21-kserve-self-signed-certs\""
Sep 18 22:46:13.055061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.055026    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:46:13.175981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.175956    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.176090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.175991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.176090 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.176017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.176190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.176099    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bqnk\" (UniqueName: \"kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.176190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.176149    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.176190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.176184    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277434    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277471    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277491    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277518 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277514    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bqnk\" (UniqueName: \"kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277828 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277535    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.277828 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277580    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.278009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277889    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.278009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277939    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.278009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.277991    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.279917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.279899    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.280182 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.280161    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.289064 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.289023    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bqnk\" (UniqueName: \"kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk\") pod \"rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") " pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.338960 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.338937    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:13.611784 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.611757    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:46:13.614415 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:46:13.614387    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod77784c2c_4071_4e0f_b712_e6d79c51fed3.slice/crio-75c69378a5e316974df9e7ff2e73ff07b125960c2ecfdbac7b1ca587260125b6 WatchSource:0}: Error finding container 75c69378a5e316974df9e7ff2e73ff07b125960c2ecfdbac7b1ca587260125b6: Status 404 returned error can't find the container with id 75c69378a5e316974df9e7ff2e73ff07b125960c2ecfdbac7b1ca587260125b6
Sep 18 22:46:13.842635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.842602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" event={"ID":"77784c2c-4071-4e0f-b712-e6d79c51fed3","Type":"ContainerStarted","Data":"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"}
Sep 18 22:46:13.842767 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.842647    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" event={"ID":"77784c2c-4071-4e0f-b712-e6d79c51fed3","Type":"ContainerStarted","Data":"75c69378a5e316974df9e7ff2e73ff07b125960c2ecfdbac7b1ca587260125b6"}
Sep 18 22:46:13.862018 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:13.861934    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" podStartSLOduration=1.861918835 podStartE2EDuration="1.861918835s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:46:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:46:13.860781753 +0000 UTC m=+1766.042366301" watchObservedRunningTime="2026-09-18 22:46:13.861918835 +0000 UTC m=+1766.043503380"
Sep 18 22:46:16.407593 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:16.407557    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:46:23.340068 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:23.340002    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:23.340455 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:23.340085    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:23.352539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:23.352513    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:23.911052 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:23.911004    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:46:30.402984 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:30.402950    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:46:39.501244 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:39.501212    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:46:45.481258 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:45.481179    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:46:46.332904 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.332868    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:46:46.337040 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.337013    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.340315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.340293    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ipp-a-e1f79c-openshift-default-dockercfg-cnf2j\""
Sep 18 22:46:46.350965 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.350943    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:46:46.449108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449077    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449117    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449275 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449135    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449375 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449266    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449375 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449375 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449363    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449384    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9nvxv\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449401    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.449502 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.449465    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.550677 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550653    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551045 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550689    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551045 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550708    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551045 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550726    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9nvxv\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551045 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551045 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.550968    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551063    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551098    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551115    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551384    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551444    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551475    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.551558 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.551528    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.552925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.552904    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.553273 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.553182    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.561323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.561297    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.561323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.561317    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9nvxv\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv\") pod \"e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") " pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.648154 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.648122    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:46.930686 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:46:46.930659    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2fdf193_4f22_4abe_9a85_a6e01eef1c93.slice/crio-0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685 WatchSource:0}: Error finding container 0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685: Status 404 returned error can't find the container with id 0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685
Sep 18 22:46:46.930968 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.930946    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:46:46.933221 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.933183    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:46:46.933291 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.933267    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:46:46.933336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:46.933314    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:46:47.031931 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:47.031880    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerStarted","Data":"345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3"}
Sep 18 22:46:47.032118 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:47.031962    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerStarted","Data":"0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685"}
Sep 18 22:46:47.056446 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:47.056397    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podStartSLOduration=1.056376607 podStartE2EDuration="1.056376607s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:46:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:46:47.053462152 +0000 UTC m=+1799.235046699" watchObservedRunningTime="2026-09-18 22:46:47.056376607 +0000 UTC m=+1799.237961156"
Sep 18 22:46:47.649361 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:47.649321    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:47.654409 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:47.654385    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:48.037620 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:48.037543    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:48.038364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:48.038345    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:46:48.387705 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:48.387622    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:46:48.390390 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:48.390364    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:46:50.360941 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:46:50.360911    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:47:00.499552 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.499509    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" podUID="77784c2c-4071-4e0f-b712-e6d79c51fed3" containerName="main" containerID="cri-o://e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6" gracePeriod=45
Sep 18 22:47:00.797266 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.797242    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:47:00.897120 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.896835    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.897305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897135    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache" (OuterVolumeSpecName: "model-cache") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:00.897305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897232    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.897305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897299    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.897498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897330    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.897498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897406    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.897498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897447    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9bqnk\" (UniqueName: \"kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk\") pod \"77784c2c-4071-4e0f-b712-e6d79c51fed3\" (UID: \"77784c2c-4071-4e0f-b712-e6d79c51fed3\") "
Sep 18 22:47:00.898328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.897939    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:00.899454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.899230    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:00.899454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.899447    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home" (OuterVolumeSpecName: "home") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:00.905524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.905388    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk" (OuterVolumeSpecName: "kube-api-access-9bqnk") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "kube-api-access-9bqnk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:47:00.906328 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.906300    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm" (OuterVolumeSpecName: "dshm") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:47:00.906498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.906335    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "77784c2c-4071-4e0f-b712-e6d79c51fed3" (UID: "77784c2c-4071-4e0f-b712-e6d79c51fed3"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:47:00.998999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.998962    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:00.998999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.998995    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77784c2c-4071-4e0f-b712-e6d79c51fed3-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:00.999235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.999005    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:00.999235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.999022    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77784c2c-4071-4e0f-b712-e6d79c51fed3-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:00.999235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:00.999057    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bqnk\" (UniqueName: \"kubernetes.io/projected/77784c2c-4071-4e0f-b712-e6d79c51fed3-kube-api-access-9bqnk\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:47:01.113376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.113282    2551 generic.go:356] "Generic (PLEG): container finished" podID="77784c2c-4071-4e0f-b712-e6d79c51fed3" containerID="e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6" exitCode=0
Sep 18 22:47:01.113376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.113348    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"
Sep 18 22:47:01.113376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.113349    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" event={"ID":"77784c2c-4071-4e0f-b712-e6d79c51fed3","Type":"ContainerDied","Data":"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"}
Sep 18 22:47:01.113611 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.113388    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt" event={"ID":"77784c2c-4071-4e0f-b712-e6d79c51fed3","Type":"ContainerDied","Data":"75c69378a5e316974df9e7ff2e73ff07b125960c2ecfdbac7b1ca587260125b6"}
Sep 18 22:47:01.113611 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.113410    2551 scope.go:117] "RemoveContainer" containerID="e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"
Sep 18 22:47:01.122341 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.122324    2551 scope.go:117] "RemoveContainer" containerID="e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"
Sep 18 22:47:01.122618 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:47:01.122598    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6\": container with ID starting with e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6 not found: ID does not exist" containerID="e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"
Sep 18 22:47:01.122665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.122627    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6"} err="failed to get container status \"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6\": rpc error: code = NotFound desc = could not find container \"e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6\": container with ID starting with e453d3d22e4f69df2ba731d6fbfd0a6b4b5a87d2aa4457228d43daa02bcc3ed6 not found: ID does not exist"
Sep 18 22:47:01.135293 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.135270    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:47:01.144500 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.144472    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w3-a91c21/rate-test-model-a91c21-kserve-5f5bff9b8c-kmwpt"]
Sep 18 22:47:01.305106 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:01.305075    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:47:02.293119 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:02.293079    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="77784c2c-4071-4e0f-b712-e6d79c51fed3" path="/var/lib/kubelet/pods/77784c2c-4071-4e0f-b712-e6d79c51fed3/volumes"
Sep 18 22:47:04.079260 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:04.079214    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Sep 18 22:47:04.079684 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:04.079291    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Sep 18 22:47:05.139168 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.139135    2551 generic.go:356] "Generic (PLEG): container finished" podID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerID="345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3" exitCode=0
Sep 18 22:47:05.139604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.139193    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerDied","Data":"345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3"}
Sep 18 22:47:05.139604 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.139526    2551 scope.go:117] "RemoveContainer" containerID="345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3"
Sep 18 22:47:05.140788 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.140755    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:47:05.140855 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.140841    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:47:05.140900 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:05.140885    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 18 22:47:06.160497 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:06.160452    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerStarted","Data":"865d50ec0ef10b0968eb4a6254ca2ab7114d2e672e498a9e22d42ae57d989014"}
Sep 18 22:47:06.649283 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:06.649247    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:47:06.649699 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:06.649668    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 18 22:47:06.649811 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:06.649739    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 18 22:47:07.648774 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:07.648735    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 18 22:47:07.649384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:07.649349    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 18 22:47:08.649343 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:08.649305    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 18 22:47:08.649904 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:08.649876    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 18 22:47:09.648922 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:09.648885    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 18 22:47:09.649232 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:09.649202    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 18 22:47:10.649141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:10.649102    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused" start-of-body=
Sep 18 22:47:10.649614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:10.649173    2551 prober.go:120] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.132.0.89:15021/healthz/ready\": dial tcp 10.132.0.89:15021: connect: connection refused"
Sep 18 22:47:11.652385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:11.652361    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:47:11.652809 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:11.652760    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:47:11.653348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:11.653328    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:47:22.135661 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.135623    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:47:22.136048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.136001    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="77784c2c-4071-4e0f-b712-e6d79c51fed3" containerName="main"
Sep 18 22:47:22.136048 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.136013    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="77784c2c-4071-4e0f-b712-e6d79c51fed3" containerName="main"
Sep 18 22:47:22.136134 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.136114    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="77784c2c-4071-4e0f-b712-e6d79c51fed3" containerName="main"
Sep 18 22:47:22.138159 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.138143    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.140818 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.140796    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-36f843\""
Sep 18 22:47:22.153363 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.153339    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:47:22.177023 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.176995    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:47:22.179382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.179362    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.181743 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.181720    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-36f843-dockercfg-72pkj\""
Sep 18 22:47:22.196834 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.196812    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:47:22.198369 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.198349    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c65g9\" (UniqueName: \"kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.198457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.198378    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.299011 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.298979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xsvxm\" (UniqueName: \"kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.299228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.299066    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.299228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.299115    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c65g9\" (UniqueName: \"kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.299228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.299138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.299769 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.299752    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.308358 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.308332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c65g9\" (UniqueName: \"kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9\") pod \"payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.400004 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.399974    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xsvxm\" (UniqueName: \"kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.400162 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.400026    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.400812 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.400792    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.410494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.410463    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xsvxm\" (UniqueName: \"kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm\") pod \"payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.449165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.449134    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:22.489553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.489516    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:22.733527 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.733502    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:47:22.735645 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:47:22.735604    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefc05940_23fd_495c_a19c_bb29a3c3a4f6.slice/crio-ac89691d7489395ebd5d105e6f54af418a8339981e67d9280c867480e487c9cb WatchSource:0}: Error finding container ac89691d7489395ebd5d105e6f54af418a8339981e67d9280c867480e487c9cb: Status 404 returned error can't find the container with id ac89691d7489395ebd5d105e6f54af418a8339981e67d9280c867480e487c9cb
Sep 18 22:47:22.792385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:22.792361    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:47:22.793921 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:47:22.793894    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ff4dcc4_35ff_4c70_9e3c_5f7633c27a70.slice/crio-9602856841e0e52a46068a674b118f2974029a7821cf5443a19731f8d991aac4 WatchSource:0}: Error finding container 9602856841e0e52a46068a674b118f2974029a7821cf5443a19731f8d991aac4: Status 404 returned error can't find the container with id 9602856841e0e52a46068a674b118f2974029a7821cf5443a19731f8d991aac4
Sep 18 22:47:23.255886 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.255845    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" event={"ID":"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70","Type":"ContainerStarted","Data":"8c0e38badb0cfd25bf04e7200a2cee78c381129cd07a4d7a22a587df81962ff1"}
Sep 18 22:47:23.256365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.255945    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:23.256365 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.255975    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" event={"ID":"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70","Type":"ContainerStarted","Data":"9602856841e0e52a46068a674b118f2974029a7821cf5443a19731f8d991aac4"}
Sep 18 22:47:23.257553 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.257527    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" event={"ID":"efc05940-23fd-495c-a19c-bb29a3c3a4f6","Type":"ContainerStarted","Data":"62b814b2ba1dd9a68055394649515ddb84d90a09416eaf355df678ec71b4c0ac"}
Sep 18 22:47:23.257666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.257578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" event={"ID":"efc05940-23fd-495c-a19c-bb29a3c3a4f6","Type":"ContainerStarted","Data":"ac89691d7489395ebd5d105e6f54af418a8339981e67d9280c867480e487c9cb"}
Sep 18 22:47:23.257666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.257607    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:23.279649 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.279602    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" podStartSLOduration=1.27958973 podStartE2EDuration="1.27958973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:47:23.276886835 +0000 UTC m=+1835.458471382" watchObservedRunningTime="2026-09-18 22:47:23.27958973 +0000 UTC m=+1835.461174274"
Sep 18 22:47:23.294051 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:23.293991    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" podStartSLOduration=1.293978402 podStartE2EDuration="1.293978402s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:47:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:47:23.292823645 +0000 UTC m=+1835.474408203" watchObservedRunningTime="2026-09-18 22:47:23.293978402 +0000 UTC m=+1835.475562947"
Sep 18 22:47:27.011380 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:27.011348    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:47:34.265455 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:34.265424    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:47:34.266026 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:34.265601    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:47:53.830296 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:53.830266    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn_6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70/payload-processing/0.log"
Sep 18 22:47:58.258597 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.258559    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:47:58.262457 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.262435    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.266020 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.265988    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e1f79c\"/\"kube-root-ca.crt\""
Sep 18 22:47:58.266187 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.266042    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e1f79c\"/\"default-dockercfg-n4tn4\""
Sep 18 22:47:58.266187 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.266026    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-ipp-a-e1f79c\"/\"openshift-service-ca.crt\""
Sep 18 22:47:58.266187 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.266026    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-ipp-a-e1f79c\"/\"ipp-route-e1f79c-kserve-self-signed-certs\""
Sep 18 22:47:58.271476 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.271453    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:47:58.316616 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316590    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.316749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316626    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.316749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316661    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jmzc9\" (UniqueName: \"kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.316749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.316749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316715    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.316956 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.316753    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417642 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417613    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417647    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417670    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jmzc9\" (UniqueName: \"kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417696    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417716    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.417782 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.417734    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.418104 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.418058    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.418188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.418166    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.418226 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.418198    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.419895 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.419875    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.420041 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.420010    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.428779 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.428757    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-jmzc9\" (UniqueName: \"kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9\") pod \"ipp-route-e1f79c-kserve-85d699d747-cbcwv\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") " pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.572690 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.572604    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:47:58.876638 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:58.876610    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:47:58.877647 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:47:58.877624    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod276de4c1_db19_453c_88d4_9fd7c39d0ce0.slice/crio-79071913c383c2aba23856dc7f62d51a95ae70690de5eda3a959c83bb0699ca8 WatchSource:0}: Error finding container 79071913c383c2aba23856dc7f62d51a95ae70690de5eda3a959c83bb0699ca8: Status 404 returned error can't find the container with id 79071913c383c2aba23856dc7f62d51a95ae70690de5eda3a959c83bb0699ca8
Sep 18 22:47:59.455308 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:59.455280    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" event={"ID":"276de4c1-db19-453c-88d4-9fd7c39d0ce0","Type":"ContainerStarted","Data":"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"}
Sep 18 22:47:59.455675 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:59.455319    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" event={"ID":"276de4c1-db19-453c-88d4-9fd7c39d0ce0","Type":"ContainerStarted","Data":"79071913c383c2aba23856dc7f62d51a95ae70690de5eda3a959c83bb0699ca8"}
Sep 18 22:47:59.474360 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:47:59.474317    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" podStartSLOduration=1.474302497 podStartE2EDuration="1.474302497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:47:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:47:59.473848188 +0000 UTC m=+1871.655432734" watchObservedRunningTime="2026-09-18 22:47:59.474302497 +0000 UTC m=+1871.655887043"
Sep 18 22:48:02.194963 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:02.194931    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:48:08.573085 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:08.573048    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:48:08.573475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:08.573116    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:48:08.585208 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:08.585180    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:48:09.521766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:09.521737    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:48:18.500477 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:18.500402    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:48:24.916171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:24.916139    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn_6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70/payload-processing/0.log"
Sep 18 22:48:25.033703 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:25.033670    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn_6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70/payload-processing/0.log"
Sep 18 22:48:33.333498 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.333463    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:48:33.337221 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.337197    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.340062 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.340041    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-f696b1-dockercfg-8bgrb\""
Sep 18 22:48:33.343862 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.343834    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-f696b1\""
Sep 18 22:48:33.375430 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.375408    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:48:33.513898 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.513865    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bln5v\" (UniqueName: \"kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.514072 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.513906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.615131 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.615061    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bln5v\" (UniqueName: \"kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.615131 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.615103    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.615796 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.615778    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.624416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.624394    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bln5v\" (UniqueName: \"kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v\") pod \"payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.647402 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.647372    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:33.921799 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:33.921773    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:48:33.923478 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:48:33.923445    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4d8acfb4_17be_4b5a_b31f_668ffdf2de12.slice/crio-225af841c0ce70215ff5174f3463f35ff5427455cb8b0bccb5af87a4ca48a9c2 WatchSource:0}: Error finding container 225af841c0ce70215ff5174f3463f35ff5427455cb8b0bccb5af87a4ca48a9c2: Status 404 returned error can't find the container with id 225af841c0ce70215ff5174f3463f35ff5427455cb8b0bccb5af87a4ca48a9c2
Sep 18 22:48:34.659528 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:34.659486    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" event={"ID":"4d8acfb4-17be-4b5a-b31f-668ffdf2de12","Type":"ContainerStarted","Data":"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"}
Sep 18 22:48:34.660158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:34.660129    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:34.660158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:34.660165    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" event={"ID":"4d8acfb4-17be-4b5a-b31f-668ffdf2de12","Type":"ContainerStarted","Data":"225af841c0ce70215ff5174f3463f35ff5427455cb8b0bccb5af87a4ca48a9c2"}
Sep 18 22:48:34.677992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:34.677936    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" podStartSLOduration=1.677919846 podStartE2EDuration="1.677919846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:48:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:48:34.677224976 +0000 UTC m=+1906.858809515" watchObservedRunningTime="2026-09-18 22:48:34.677919846 +0000 UTC m=+1906.859504397"
Sep 18 22:48:37.630992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:37.630965    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:48:45.667447 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:45.667419    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:48:48.716449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:48.716421    2551 scope.go:117] "RemoveContainer" containerID="059bc860b7ac50fdc066bb92b46d98da1fd2bf6bd1c284b30c79f6761f78e450"
Sep 18 22:48:48.725206 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:48.725187    2551 scope.go:117] "RemoveContainer" containerID="b0920a6b170351d6a6d31a77d91b0559e2c60f0adf6b205af6b2fa9d9c266369"
Sep 18 22:48:59.808829 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.808798    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"]
Sep 18 22:48:59.812384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.812369    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:48:59.814929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.814910    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:48:59.815065 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.814909    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:48:59.821494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.821469    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"]
Sep 18 22:48:59.843969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.843935    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t\" (UID: \"17fd661e-ae63-4dbd-afe4-9ff2d6872574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:48:59.945167 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.945137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t\" (UID: \"17fd661e-ae63-4dbd-afe4-9ff2d6872574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:48:59.945725 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:48:59.945704    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t\" (UID: \"17fd661e-ae63-4dbd-afe4-9ff2d6872574\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:49:00.123683 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:00.123594    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:49:00.400171 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:00.400147    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"]
Sep 18 22:49:00.402821 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:49:00.402799    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17fd661e_ae63_4dbd_afe4_9ff2d6872574.slice/crio-9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420 WatchSource:0}: Error finding container 9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420: Status 404 returned error can't find the container with id 9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420
Sep 18 22:49:00.805637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:00.805599    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t" event={"ID":"17fd661e-ae63-4dbd-afe4-9ff2d6872574","Type":"ContainerStarted","Data":"b56a2e703f54b49bb893276d9ead786f46a3c6039dcd2ed5e3a77af7d9265d43"}
Sep 18 22:49:00.805786 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:00.805651    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t" event={"ID":"17fd661e-ae63-4dbd-afe4-9ff2d6872574","Type":"ContainerStarted","Data":"9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420"}
Sep 18 22:49:00.823086 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:00.823041    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t" podStartSLOduration=1.823013187 podStartE2EDuration="1.823013187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:49:00.820902695 +0000 UTC m=+1933.002487241" watchObservedRunningTime="2026-09-18 22:49:00.823013187 +0000 UTC m=+1933.004597732"
Sep 18 22:49:01.812280 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:01.812245    2551 generic.go:356] "Generic (PLEG): container finished" podID="17fd661e-ae63-4dbd-afe4-9ff2d6872574" containerID="b56a2e703f54b49bb893276d9ead786f46a3c6039dcd2ed5e3a77af7d9265d43" exitCode=0
Sep 18 22:49:01.812461 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:01.812337    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t" event={"ID":"17fd661e-ae63-4dbd-afe4-9ff2d6872574","Type":"ContainerDied","Data":"b56a2e703f54b49bb893276d9ead786f46a3c6039dcd2ed5e3a77af7d9265d43"}
Sep 18 22:49:02.945821 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:02.945799    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:49:03.073490 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.073429    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca\") pod \"17fd661e-ae63-4dbd-afe4-9ff2d6872574\" (UID: \"17fd661e-ae63-4dbd-afe4-9ff2d6872574\") "
Sep 18 22:49:03.073749 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.073727    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "17fd661e-ae63-4dbd-afe4-9ff2d6872574" (UID: "17fd661e-ae63-4dbd-afe4-9ff2d6872574"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:49:03.174187 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.174159    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/17fd661e-ae63-4dbd-afe4-9ff2d6872574-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:03.824594 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.824555    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"
Sep 18 22:49:03.824772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.824595    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t" event={"ID":"17fd661e-ae63-4dbd-afe4-9ff2d6872574","Type":"ContainerDied","Data":"9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420"}
Sep 18 22:49:03.824772 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:03.824654    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9f72dfc7e2243075fbc8bf5e369750f8aa41dcdfb1017177879058fccb4f9420"
Sep 18 22:49:10.860228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:10.857420    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:49:10.860228 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:10.857764    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" podUID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" containerName="payload-pre-processing" containerID="cri-o://cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b" gracePeriod=30
Sep 18 22:49:11.020053 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.020000    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"]
Sep 18 22:49:11.022912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.022887    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-f696b1-7mg6t"]
Sep 18 22:49:11.125442 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.125382    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:49:11.241930 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.241901    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume\") pod \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") "
Sep 18 22:49:11.242099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.242008    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-bln5v\" (UniqueName: \"kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v\") pod \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\" (UID: \"4d8acfb4-17be-4b5a-b31f-668ffdf2de12\") "
Sep 18 22:49:11.242268 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.242246    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4d8acfb4-17be-4b5a-b31f-668ffdf2de12" (UID: "4d8acfb4-17be-4b5a-b31f-668ffdf2de12"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:49:11.244158 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.244138    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v" (OuterVolumeSpecName: "kube-api-access-bln5v") pod "4d8acfb4-17be-4b5a-b31f-668ffdf2de12" (UID: "4d8acfb4-17be-4b5a-b31f-668ffdf2de12"). InnerVolumeSpecName "kube-api-access-bln5v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:49:11.343564 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.343539    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:11.343564 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.343561    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bln5v\" (UniqueName: \"kubernetes.io/projected/4d8acfb4-17be-4b5a-b31f-668ffdf2de12-kube-api-access-bln5v\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:11.883577 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.883542    2551 generic.go:356] "Generic (PLEG): container finished" podID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" containerID="cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b" exitCode=0
Sep 18 22:49:11.883577 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.883580    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"
Sep 18 22:49:11.884112 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.883632    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" event={"ID":"4d8acfb4-17be-4b5a-b31f-668ffdf2de12","Type":"ContainerDied","Data":"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"}
Sep 18 22:49:11.884112 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.883697    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq" event={"ID":"4d8acfb4-17be-4b5a-b31f-668ffdf2de12","Type":"ContainerDied","Data":"225af841c0ce70215ff5174f3463f35ff5427455cb8b0bccb5af87a4ca48a9c2"}
Sep 18 22:49:11.884112 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.883733    2551 scope.go:117] "RemoveContainer" containerID="cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"
Sep 18 22:49:11.893142 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.893126    2551 scope.go:117] "RemoveContainer" containerID="cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"
Sep 18 22:49:11.893419 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:49:11.893398    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b\": container with ID starting with cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b not found: ID does not exist" containerID="cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"
Sep 18 22:49:11.893481 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.893432    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b"} err="failed to get container status \"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b\": rpc error: code = NotFound desc = could not find container \"cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b\": container with ID starting with cdc2a5268a4f24b768e0f8116b3cdb16f8860b24f0ef536547f64775fea7d03b not found: ID does not exist"
Sep 18 22:49:11.904831 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.904806    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:49:11.908891 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:11.908870    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-f696b1-57d88c468b-26pmq"]
Sep 18 22:49:12.291371 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:12.291342    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="17fd661e-ae63-4dbd-afe4-9ff2d6872574" path="/var/lib/kubelet/pods/17fd661e-ae63-4dbd-afe4-9ff2d6872574/volumes"
Sep 18 22:49:12.291697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:12.291684    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" path="/var/lib/kubelet/pods/4d8acfb4-17be-4b5a-b31f-668ffdf2de12/volumes"
Sep 18 22:49:15.747966 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:15.747933    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:49:21.664637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.664606    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"]
Sep 18 22:49:21.665060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.664998    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="17fd661e-ae63-4dbd-afe4-9ff2d6872574" containerName="revoke-keys"
Sep 18 22:49:21.665060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.665010    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="17fd661e-ae63-4dbd-afe4-9ff2d6872574" containerName="revoke-keys"
Sep 18 22:49:21.665060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.665024    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" containerName="payload-pre-processing"
Sep 18 22:49:21.665060 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.665043    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" containerName="payload-pre-processing"
Sep 18 22:49:21.665230 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.665114    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="17fd661e-ae63-4dbd-afe4-9ff2d6872574" containerName="revoke-keys"
Sep 18 22:49:21.665230 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.665125    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4d8acfb4-17be-4b5a-b31f-668ffdf2de12" containerName="payload-pre-processing"
Sep 18 22:49:21.669588 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.669572    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:21.671936 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.671905    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:49:21.672077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.672012    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:49:21.679605 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.679574    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"]
Sep 18 22:49:21.823565 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.823531    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t\" (UID: \"92d0cba8-f364-49ab-9c67-396a13c3588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:21.923945 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.923869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t\" (UID: \"92d0cba8-f364-49ab-9c67-396a13c3588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:21.924417 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.924397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t\" (UID: \"92d0cba8-f364-49ab-9c67-396a13c3588a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:21.981845 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:21.981820    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:22.274107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:22.274013    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"]
Sep 18 22:49:22.277069 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:49:22.277039    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92d0cba8_f364_49ab_9c67_396a13c3588a.slice/crio-02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f WatchSource:0}: Error finding container 02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f: Status 404 returned error can't find the container with id 02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f
Sep 18 22:49:22.948200 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:22.948172    2551 generic.go:356] "Generic (PLEG): container finished" podID="92d0cba8-f364-49ab-9c67-396a13c3588a" containerID="07b1c9bfda875304e113c540b7a04b2ba566e2f21ca12000902fdb9299e45270" exitCode=0
Sep 18 22:49:22.948500 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:22.948255    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t" event={"ID":"92d0cba8-f364-49ab-9c67-396a13c3588a","Type":"ContainerDied","Data":"07b1c9bfda875304e113c540b7a04b2ba566e2f21ca12000902fdb9299e45270"}
Sep 18 22:49:22.948500 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:22.948309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t" event={"ID":"92d0cba8-f364-49ab-9c67-396a13c3588a","Type":"ContainerStarted","Data":"02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f"}
Sep 18 22:49:24.089179 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.089154    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:24.244382 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.244313    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca\") pod \"92d0cba8-f364-49ab-9c67-396a13c3588a\" (UID: \"92d0cba8-f364-49ab-9c67-396a13c3588a\") "
Sep 18 22:49:24.244701 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.244678    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "92d0cba8-f364-49ab-9c67-396a13c3588a" (UID: "92d0cba8-f364-49ab-9c67-396a13c3588a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:49:24.345578 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.345546    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/92d0cba8-f364-49ab-9c67-396a13c3588a-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:24.960816 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.960780    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t" event={"ID":"92d0cba8-f364-49ab-9c67-396a13c3588a","Type":"ContainerDied","Data":"02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f"}
Sep 18 22:49:24.960981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.960832    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02d6abdd5e5aea27d1bcbaa0190fd63a9f45bb3d7c779b717889f7fdd45b411f"
Sep 18 22:49:24.960981 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:24.960792    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"
Sep 18 22:49:34.899395 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:34.899360    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:49:37.723887 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:37.723857    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"]
Sep 18 22:49:37.729217 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:37.729191    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-e1f79c-b272t"]
Sep 18 22:49:38.290977 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:38.290947    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="92d0cba8-f364-49ab-9c67-396a13c3588a" path="/var/lib/kubelet/pods/92d0cba8-f364-49ab-9c67-396a13c3588a/volumes"
Sep 18 22:49:43.070341 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:43.070247    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:49:47.894293 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:47.894253    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:49:48.749143 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:48.749110    2551 scope.go:117] "RemoveContainer" containerID="9829f767daf4031d17e348fa0f3e1bcc0c6fe7a71e94e93457804cb9f9b0fef1"
Sep 18 22:49:48.758099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:48.758079    2551 scope.go:117] "RemoveContainer" containerID="e8416e9e042fbe33e6396f84b649f5d9f9f7f7cc0eaa350925f3a3331e45d832"
Sep 18 22:49:48.766067 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:48.766042    2551 scope.go:117] "RemoveContainer" containerID="7c1d2cb183a45e52adbb83e37521a670498c2fdea00d2a7d05c31848b22454fb"
Sep 18 22:49:48.775535 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:48.775508    2551 scope.go:117] "RemoveContainer" containerID="972040a58b263c50062729b2978a80b72338a2502857e59ec7139371a2cd10ae"
Sep 18 22:49:58.089737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.089671    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" podUID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" containerName="main" containerID="cri-o://8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d" gracePeriod=45
Sep 18 22:49:58.345077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.345009    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:49:58.434795 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434761    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.434970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434807    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.434970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434829    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-jmzc9\" (UniqueName: \"kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.434970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434900    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.434970 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434933    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.435217 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.434989    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs\") pod \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\" (UID: \"276de4c1-db19-453c-88d4-9fd7c39d0ce0\") "
Sep 18 22:49:58.435217 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.435080    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:58.435217 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.435092    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache" (OuterVolumeSpecName: "model-cache") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:58.435345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.435234    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home" (OuterVolumeSpecName: "home") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:58.435345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.435295    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tmp-dir\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:58.435345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.435310    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-model-cache\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:58.437214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.437190    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9" (OuterVolumeSpecName: "kube-api-access-jmzc9") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "kube-api-access-jmzc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:49:58.437214 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.437198    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm" (OuterVolumeSpecName: "dshm") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:49:58.437345 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.437198    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "276de4c1-db19-453c-88d4-9fd7c39d0ce0" (UID: "276de4c1-db19-453c-88d4-9fd7c39d0ce0"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 18 22:49:58.536599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.536553    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/276de4c1-db19-453c-88d4-9fd7c39d0ce0-tls-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:58.536599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.536597    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jmzc9\" (UniqueName: \"kubernetes.io/projected/276de4c1-db19-453c-88d4-9fd7c39d0ce0-kube-api-access-jmzc9\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:58.536599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.536607    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-dshm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:58.536839 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:58.536616    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/276de4c1-db19-453c-88d4-9fd7c39d0ce0-home\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:49:59.146234 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.146201    2551 generic.go:356] "Generic (PLEG): container finished" podID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" containerID="8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d" exitCode=0
Sep 18 22:49:59.146647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.146244    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" event={"ID":"276de4c1-db19-453c-88d4-9fd7c39d0ce0","Type":"ContainerDied","Data":"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"}
Sep 18 22:49:59.146647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.146274    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv" event={"ID":"276de4c1-db19-453c-88d4-9fd7c39d0ce0","Type":"ContainerDied","Data":"79071913c383c2aba23856dc7f62d51a95ae70690de5eda3a959c83bb0699ca8"}
Sep 18 22:49:59.146647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.146291    2551 scope.go:117] "RemoveContainer" containerID="8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"
Sep 18 22:49:59.146647 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.146247    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"
Sep 18 22:49:59.155533 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.155517    2551 scope.go:117] "RemoveContainer" containerID="8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"
Sep 18 22:49:59.155804 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:49:59.155785    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d\": container with ID starting with 8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d not found: ID does not exist" containerID="8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"
Sep 18 22:49:59.155863 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.155815    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d"} err="failed to get container status \"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d\": rpc error: code = NotFound desc = could not find container \"8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d\": container with ID starting with 8a4cc7137f014b0640ed7bc7cfa78c06d4cf171acec0df7b0ed29d1ad039ce2d not found: ID does not exist"
Sep 18 22:49:59.168277 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.168253    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:49:59.174398 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:49:59.174380    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-ipp-a-e1f79c/ipp-route-e1f79c-kserve-85d699d747-cbcwv"]
Sep 18 22:50:00.296054 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:00.295992    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" path="/var/lib/kubelet/pods/276de4c1-db19-453c-88d4-9fd7c39d0ce0/volumes"
Sep 18 22:50:41.159834 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.159802    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:50:41.160342 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.160046    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" containerID="cri-o://865d50ec0ef10b0968eb4a6254ca2ab7114d2e672e498a9e22d42ae57d989014" gracePeriod=30
Sep 18 22:50:41.527318 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527221    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"]
Sep 18 22:50:41.527738 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527719    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" containerName="main"
Sep 18 22:50:41.527835 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527741    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" containerName="main"
Sep 18 22:50:41.527835 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527765    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="92d0cba8-f364-49ab-9c67-396a13c3588a" containerName="revoke-keys"
Sep 18 22:50:41.527835 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527773    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="92d0cba8-f364-49ab-9c67-396a13c3588a" containerName="revoke-keys"
Sep 18 22:50:41.527990 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527861    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="276de4c1-db19-453c-88d4-9fd7c39d0ce0" containerName="main"
Sep 18 22:50:41.527990 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.527875    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="92d0cba8-f364-49ab-9c67-396a13c3588a" containerName="revoke-keys"
Sep 18 22:50:41.530760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.530739    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:41.533361 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.533334    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:50:41.533474 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.533345    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:50:41.540304 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.540275    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"]
Sep 18 22:50:41.608588 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.608550    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m\" (UID: \"24f3e843-cdd0-4330-ba7a-344e93651bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:41.655891 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.655858    2551 patch_prober.go:28] interesting pod/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Sep 18 22:50:41.656061 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.655928    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Sep 18 22:50:41.709917 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.709890    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m\" (UID: \"24f3e843-cdd0-4330-ba7a-344e93651bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:41.710534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.710514    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m\" (UID: \"24f3e843-cdd0-4330-ba7a-344e93651bad\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:41.842741 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:41.842632    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:42.123470 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:42.123397    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"]
Sep 18 22:50:42.125991 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:50:42.125964    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24f3e843_cdd0_4330_ba7a_344e93651bad.slice/crio-dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33 WatchSource:0}: Error finding container dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33: Status 404 returned error can't find the container with id dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33
Sep 18 22:50:42.375599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:42.375513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m" event={"ID":"24f3e843-cdd0-4330-ba7a-344e93651bad","Type":"ContainerStarted","Data":"dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33"}
Sep 18 22:50:43.382145 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:43.382017    2551 generic.go:356] "Generic (PLEG): container finished" podID="24f3e843-cdd0-4330-ba7a-344e93651bad" containerID="2bcbc8428764e68549e9630c536d54b2c77bf60748b5eaa59e48b0ffa2d6c0ec" exitCode=0
Sep 18 22:50:43.382145 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:43.382103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m" event={"ID":"24f3e843-cdd0-4330-ba7a-344e93651bad","Type":"ContainerDied","Data":"2bcbc8428764e68549e9630c536d54b2c77bf60748b5eaa59e48b0ffa2d6c0ec"}
Sep 18 22:50:44.527099 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:44.527073    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:44.633993 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:44.633963    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca\") pod \"24f3e843-cdd0-4330-ba7a-344e93651bad\" (UID: \"24f3e843-cdd0-4330-ba7a-344e93651bad\") "
Sep 18 22:50:44.634332 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:44.634309    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "24f3e843-cdd0-4330-ba7a-344e93651bad" (UID: "24f3e843-cdd0-4330-ba7a-344e93651bad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:50:44.735416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:44.735385    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24f3e843-cdd0-4330-ba7a-344e93651bad-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:45.394603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:45.394575    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"
Sep 18 22:50:45.394603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:45.394596    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m" event={"ID":"24f3e843-cdd0-4330-ba7a-344e93651bad","Type":"ContainerDied","Data":"dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33"}
Sep 18 22:50:45.394804 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:45.394631    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="dd4fe1bcf7d660f89a08084cb8ef8dbb171098b7eca0a27e45c852a353501f33"
Sep 18 22:50:46.402437 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.402411    2551 generic.go:356] "Generic (PLEG): container finished" podID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerID="865d50ec0ef10b0968eb4a6254ca2ab7114d2e672e498a9e22d42ae57d989014" exitCode=0
Sep 18 22:50:46.402760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.402479    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerDied","Data":"865d50ec0ef10b0968eb4a6254ca2ab7114d2e672e498a9e22d42ae57d989014"}
Sep 18 22:50:46.402760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.402548    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz" event={"ID":"e2fdf193-4f22-4abe-9a85-a6e01eef1c93","Type":"ContainerDied","Data":"0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685"}
Sep 18 22:50:46.402760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.402577    2551 scope.go:117] "RemoveContainer" containerID="345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3"
Sep 18 22:50:46.402760 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.402581    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0471d2f75c3a41405f0885a97d0e77bf4c53d8ec77c8da2b9f145e40d21a9685"
Sep 18 22:50:46.415229 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:50:46.415212    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3\": container with ID starting with 345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3 not found: ID does not exist" containerID="345e0816b0be77caeb6c2041ecccc128fb54cc72803bd29e4a912043cf91ddf3"
Sep 18 22:50:46.418753 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.418735    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:50:46.452732 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452681    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452824 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452736    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9nvxv\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452824 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452777    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452824 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452810    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452861    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452901    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452927    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452955    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.452992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.452957    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data" (OuterVolumeSpecName: "istio-data") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:50:46.453269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453013    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket\") pod \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\" (UID: \"e2fdf193-4f22-4abe-9a85-a6e01eef1c93\") "
Sep 18 22:50:46.453269 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453208    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:50:46.453435 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453398    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:50:46.453557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453431    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:50:46.453557 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453546    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.453697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453566    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.453697 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.453567    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:50:46.455174 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.455146    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:50:46.455289 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.455213    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv" (OuterVolumeSpecName: "kube-api-access-9nvxv") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "kube-api-access-9nvxv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:50:46.455412 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.455393    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token" (OuterVolumeSpecName: "istio-token") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:50:46.455475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.455417    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "e2fdf193-4f22-4abe-9a85-a6e01eef1c93" (UID: "e2fdf193-4f22-4abe-9a85-a6e01eef1c93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:50:46.554224 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554203    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554224 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554223    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554233    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9nvxv\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-kube-api-access-9nvxv\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554242    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554250    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554257    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:46.554349 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:46.554265    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e2fdf193-4f22-4abe-9a85-a6e01eef1c93-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:47.416920 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:47.416886    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"
Sep 18 22:50:47.439996 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:47.439972    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:50:47.444809 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:47.444788    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ipp-a-e1f79c-openshift-default-6b94c7566d-cvdfz"]
Sep 18 22:50:48.290913 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:48.290885    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" path="/var/lib/kubelet/pods/e2fdf193-4f22-4abe-9a85-a6e01eef1c93/volumes"
Sep 18 22:50:52.201277 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.201239    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:50:52.201702 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.201532    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" podUID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" containerName="payload-processing" containerID="cri-o://8c0e38badb0cfd25bf04e7200a2cee78c381129cd07a4d7a22a587df81962ff1" gracePeriod=30
Sep 18 22:50:52.210220 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.210195    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:50:52.210421 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.210400    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" podUID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" containerName="payload-pre-processing" containerID="cri-o://62b814b2ba1dd9a68055394649515ddb84d90a09416eaf355df678ec71b4c0ac" gracePeriod=30
Sep 18 22:50:52.446150 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.446122    2551 generic.go:356] "Generic (PLEG): container finished" podID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" containerID="8c0e38badb0cfd25bf04e7200a2cee78c381129cd07a4d7a22a587df81962ff1" exitCode=0
Sep 18 22:50:52.446326 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.446197    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" event={"ID":"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70","Type":"ContainerDied","Data":"8c0e38badb0cfd25bf04e7200a2cee78c381129cd07a4d7a22a587df81962ff1"}
Sep 18 22:50:52.448108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.448087    2551 generic.go:356] "Generic (PLEG): container finished" podID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" containerID="62b814b2ba1dd9a68055394649515ddb84d90a09416eaf355df678ec71b4c0ac" exitCode=0
Sep 18 22:50:52.448235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.448161    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" event={"ID":"efc05940-23fd-495c-a19c-bb29a3c3a4f6","Type":"ContainerDied","Data":"62b814b2ba1dd9a68055394649515ddb84d90a09416eaf355df678ec71b4c0ac"}
Sep 18 22:50:52.491289 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.491267    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:50:52.494165 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.494147    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:50:52.510564 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.510544    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"]
Sep 18 22:50:52.523524 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.523500    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-36f843-8vx8m"]
Sep 18 22:50:52.608734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.608709    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume\") pod \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") "
Sep 18 22:50:52.608895 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.608769    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c65g9\" (UniqueName: \"kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9\") pod \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") "
Sep 18 22:50:52.608895 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.608811    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume\") pod \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\" (UID: \"efc05940-23fd-495c-a19c-bb29a3c3a4f6\") "
Sep 18 22:50:52.608895 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.608855    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xsvxm\" (UniqueName: \"kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm\") pod \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\" (UID: \"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70\") "
Sep 18 22:50:52.609172 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.609150    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" (UID: "6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:50:52.609236 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.609190    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "efc05940-23fd-495c-a19c-bb29a3c3a4f6" (UID: "efc05940-23fd-495c-a19c-bb29a3c3a4f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:50:52.610968 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.610948    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm" (OuterVolumeSpecName: "kube-api-access-xsvxm") pod "6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" (UID: "6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70"). InnerVolumeSpecName "kube-api-access-xsvxm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:50:52.611021 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.610976    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9" (OuterVolumeSpecName: "kube-api-access-c65g9") pod "efc05940-23fd-495c-a19c-bb29a3c3a4f6" (UID: "efc05940-23fd-495c-a19c-bb29a3c3a4f6"). InnerVolumeSpecName "kube-api-access-c65g9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:50:52.710370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.710315    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:52.710370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.710337    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c65g9\" (UniqueName: \"kubernetes.io/projected/efc05940-23fd-495c-a19c-bb29a3c3a4f6-kube-api-access-c65g9\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:52.710370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.710346    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/efc05940-23fd-495c-a19c-bb29a3c3a4f6-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:52.710370 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:52.710354    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xsvxm\" (UniqueName: \"kubernetes.io/projected/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70-kube-api-access-xsvxm\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:50:53.455305 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.455280    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"
Sep 18 22:50:53.455710 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.455314    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn" event={"ID":"6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70","Type":"ContainerDied","Data":"9602856841e0e52a46068a674b118f2974029a7821cf5443a19731f8d991aac4"}
Sep 18 22:50:53.455710 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.455380    2551 scope.go:117] "RemoveContainer" containerID="8c0e38badb0cfd25bf04e7200a2cee78c381129cd07a4d7a22a587df81962ff1"
Sep 18 22:50:53.456883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.456840    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"
Sep 18 22:50:53.456883 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.456865    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd" event={"ID":"efc05940-23fd-495c-a19c-bb29a3c3a4f6","Type":"ContainerDied","Data":"ac89691d7489395ebd5d105e6f54af418a8339981e67d9280c867480e487c9cb"}
Sep 18 22:50:53.465391 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.465378    2551 scope.go:117] "RemoveContainer" containerID="62b814b2ba1dd9a68055394649515ddb84d90a09416eaf355df678ec71b4c0ac"
Sep 18 22:50:53.504736 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.504706    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:50:53.509464 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.509445    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-36f843-586fd4f867-qpvcn"]
Sep 18 22:50:53.533059 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.533024    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:50:53.539212 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:53.539189    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-36f843-749fcd446b-ngwsd"]
Sep 18 22:50:54.290767 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:54.290735    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="24f3e843-cdd0-4330-ba7a-344e93651bad" path="/var/lib/kubelet/pods/24f3e843-cdd0-4330-ba7a-344e93651bad/volumes"
Sep 18 22:50:54.291161 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:54.291147    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" path="/var/lib/kubelet/pods/6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70/volumes"
Sep 18 22:50:54.291462 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:54.291451    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" path="/var/lib/kubelet/pods/efc05940-23fd-495c-a19c-bb29a3c3a4f6/volumes"
Sep 18 22:50:55.387384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:55.387351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:50:59.398559 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.398522    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"]
Sep 18 22:50:59.399108 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399086    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" containerName="payload-processing"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399110    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" containerName="payload-processing"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399138    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399147    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399161    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" containerName="payload-pre-processing"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399170    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" containerName="payload-pre-processing"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399189    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="24f3e843-cdd0-4330-ba7a-344e93651bad" containerName="revoke-keys"
Sep 18 22:50:59.399195 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399198    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="24f3e843-cdd0-4330-ba7a-344e93651bad" containerName="revoke-keys"
Sep 18 22:50:59.399561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399282    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6ff4dcc4-35ff-4c70-9e3c-5f7633c27a70" containerName="payload-processing"
Sep 18 22:50:59.399561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399294    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:50:59.399561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399307    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="24f3e843-cdd0-4330-ba7a-344e93651bad" containerName="revoke-keys"
Sep 18 22:50:59.399561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399317    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:50:59.399561 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.399329    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="efc05940-23fd-495c-a19c-bb29a3c3a4f6" containerName="payload-pre-processing"
Sep 18 22:50:59.403823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.403800    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:50:59.407103 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.407085    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:50:59.407307 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.407295    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:50:59.440589 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.440566    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"]
Sep 18 22:50:59.464506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.464470    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl\" (UID: \"c5aa114e-3bd1-4473-8391-31a774c2bf20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:50:59.565908 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.565883    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl\" (UID: \"c5aa114e-3bd1-4473-8391-31a774c2bf20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:50:59.566463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.566446    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl\" (UID: \"c5aa114e-3bd1-4473-8391-31a774c2bf20\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:50:59.714274 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:50:59.714203    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:51:00.005229 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:00.005203    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"]
Sep 18 22:51:00.007101 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:51:00.007068    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5aa114e_3bd1_4473_8391_31a774c2bf20.slice/crio-e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee WatchSource:0}: Error finding container e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee: Status 404 returned error can't find the container with id e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee
Sep 18 22:51:00.503111 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:00.503078    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl" event={"ID":"c5aa114e-3bd1-4473-8391-31a774c2bf20","Type":"ContainerStarted","Data":"256ecc4bcc0304f5452e0a83cb621977c52594f3660b20f8cc2eed2e86d97e2f"}
Sep 18 22:51:00.503432 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:00.503133    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl" event={"ID":"c5aa114e-3bd1-4473-8391-31a774c2bf20","Type":"ContainerStarted","Data":"e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee"}
Sep 18 22:51:00.524792 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:00.524750    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl" podStartSLOduration=1.524734353 podStartE2EDuration="1.524734353s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:50:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:51:00.523395415 +0000 UTC m=+2052.704979961" watchObservedRunningTime="2026-09-18 22:51:00.524734353 +0000 UTC m=+2052.706318899"
Sep 18 22:51:01.509873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:01.509839    2551 generic.go:356] "Generic (PLEG): container finished" podID="c5aa114e-3bd1-4473-8391-31a774c2bf20" containerID="256ecc4bcc0304f5452e0a83cb621977c52594f3660b20f8cc2eed2e86d97e2f" exitCode=0
Sep 18 22:51:01.510256 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:01.509884    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl" event={"ID":"c5aa114e-3bd1-4473-8391-31a774c2bf20","Type":"ContainerDied","Data":"256ecc4bcc0304f5452e0a83cb621977c52594f3660b20f8cc2eed2e86d97e2f"}
Sep 18 22:51:02.648912 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:02.648887    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:51:02.794410 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:02.794341    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca\") pod \"c5aa114e-3bd1-4473-8391-31a774c2bf20\" (UID: \"c5aa114e-3bd1-4473-8391-31a774c2bf20\") "
Sep 18 22:51:02.794666 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:02.794645    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c5aa114e-3bd1-4473-8391-31a774c2bf20" (UID: "c5aa114e-3bd1-4473-8391-31a774c2bf20"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:02.895729 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:02.895705    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c5aa114e-3bd1-4473-8391-31a774c2bf20-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:03.521784 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:03.521752    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"
Sep 18 22:51:03.521944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:03.521784    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl" event={"ID":"c5aa114e-3bd1-4473-8391-31a774c2bf20","Type":"ContainerDied","Data":"e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee"}
Sep 18 22:51:03.521944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:03.521833    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e72a11940f0754be51fe4ee5f6b46b9066094fd7e6129c8c4a5b3ec0b10c17ee"
Sep 18 22:51:09.890693 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:09.890650    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:51:09.891150 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:09.890912    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" podUID="ee186927-348d-496a-bfe8-9b67a48d5ee4" containerName="payload-processing" containerID="cri-o://2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5" gracePeriod=30
Sep 18 22:51:09.898767 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:09.898716    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:51:09.899842 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:09.899600    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" podUID="742d9377-2264-4c69-92d3-b005cf092233" containerName="payload-pre-processing" containerID="cri-o://abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4" gracePeriod=30
Sep 18 22:51:10.079635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.079596    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"]
Sep 18 22:51:10.083520 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.083486    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-d6c0ac-h6tbl"]
Sep 18 22:51:10.174102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.174076    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:51:10.177039 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.177007    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:51:10.252127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252099    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume\") pod \"742d9377-2264-4c69-92d3-b005cf092233\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") "
Sep 18 22:51:10.252294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252170    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vcmf\" (UniqueName: \"kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf\") pod \"742d9377-2264-4c69-92d3-b005cf092233\" (UID: \"742d9377-2264-4c69-92d3-b005cf092233\") "
Sep 18 22:51:10.252294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252216    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5xj6l\" (UniqueName: \"kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l\") pod \"ee186927-348d-496a-bfe8-9b67a48d5ee4\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") "
Sep 18 22:51:10.252294 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252251    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume\") pod \"ee186927-348d-496a-bfe8-9b67a48d5ee4\" (UID: \"ee186927-348d-496a-bfe8-9b67a48d5ee4\") "
Sep 18 22:51:10.252598 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252558    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "742d9377-2264-4c69-92d3-b005cf092233" (UID: "742d9377-2264-4c69-92d3-b005cf092233"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:10.252675 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.252626    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ee186927-348d-496a-bfe8-9b67a48d5ee4" (UID: "ee186927-348d-496a-bfe8-9b67a48d5ee4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:10.254405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.254378    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf" (OuterVolumeSpecName: "kube-api-access-9vcmf") pod "742d9377-2264-4c69-92d3-b005cf092233" (UID: "742d9377-2264-4c69-92d3-b005cf092233"). InnerVolumeSpecName "kube-api-access-9vcmf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:10.254405 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.254401    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l" (OuterVolumeSpecName: "kube-api-access-5xj6l") pod "ee186927-348d-496a-bfe8-9b67a48d5ee4" (UID: "ee186927-348d-496a-bfe8-9b67a48d5ee4"). InnerVolumeSpecName "kube-api-access-5xj6l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:10.291357 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.291321    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c5aa114e-3bd1-4473-8391-31a774c2bf20" path="/var/lib/kubelet/pods/c5aa114e-3bd1-4473-8391-31a774c2bf20/volumes"
Sep 18 22:51:10.352952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.352920    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5xj6l\" (UniqueName: \"kubernetes.io/projected/ee186927-348d-496a-bfe8-9b67a48d5ee4-kube-api-access-5xj6l\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:10.352952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.352948    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee186927-348d-496a-bfe8-9b67a48d5ee4-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:10.352952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.352957    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/742d9377-2264-4c69-92d3-b005cf092233-plugins-config-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:10.353210 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.352966    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vcmf\" (UniqueName: \"kubernetes.io/projected/742d9377-2264-4c69-92d3-b005cf092233-kube-api-access-9vcmf\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:10.558600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.558511    2551 generic.go:356] "Generic (PLEG): container finished" podID="742d9377-2264-4c69-92d3-b005cf092233" containerID="abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4" exitCode=0
Sep 18 22:51:10.558600 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.558556    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"
Sep 18 22:51:10.558813 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.558592    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" event={"ID":"742d9377-2264-4c69-92d3-b005cf092233","Type":"ContainerDied","Data":"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"}
Sep 18 22:51:10.558813 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.558648    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz" event={"ID":"742d9377-2264-4c69-92d3-b005cf092233","Type":"ContainerDied","Data":"b1970465e6280fa7b296b5ed81b47ae00302eede818b77d5d86d6ba35a4e92ff"}
Sep 18 22:51:10.558813 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.558678    2551 scope.go:117] "RemoveContainer" containerID="abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"
Sep 18 22:51:10.560016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.559994    2551 generic.go:356] "Generic (PLEG): container finished" podID="ee186927-348d-496a-bfe8-9b67a48d5ee4" containerID="2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5" exitCode=0
Sep 18 22:51:10.560128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.560019    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" event={"ID":"ee186927-348d-496a-bfe8-9b67a48d5ee4","Type":"ContainerDied","Data":"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"}
Sep 18 22:51:10.560128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.560070    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"
Sep 18 22:51:10.560128 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.560090    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb" event={"ID":"ee186927-348d-496a-bfe8-9b67a48d5ee4","Type":"ContainerDied","Data":"8054179b237896fefab912447aa6b7f69af1c8467848215f84a04d2b891712ac"}
Sep 18 22:51:10.567495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.567466    2551 scope.go:117] "RemoveContainer" containerID="abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"
Sep 18 22:51:10.567828 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:51:10.567809    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4\": container with ID starting with abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4 not found: ID does not exist" containerID="abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"
Sep 18 22:51:10.567907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.567836    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4"} err="failed to get container status \"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4\": rpc error: code = NotFound desc = could not find container \"abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4\": container with ID starting with abcc0bb476b6d3bee5367f7e72e40a851f57e7864f480f919163b9de8809bfd4 not found: ID does not exist"
Sep 18 22:51:10.567907 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.567856    2551 scope.go:117] "RemoveContainer" containerID="2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"
Sep 18 22:51:10.576190 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.575737    2551 scope.go:117] "RemoveContainer" containerID="2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"
Sep 18 22:51:10.576453 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:51:10.576423    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5\": container with ID starting with 2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5 not found: ID does not exist" containerID="2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"
Sep 18 22:51:10.576569 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.576464    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5"} err="failed to get container status \"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5\": rpc error: code = NotFound desc = could not find container \"2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5\": container with ID starting with 2c9272425d5b950c703a1417485a5b5248e85ad6fd461cfcb13d44f39ef658b5 not found: ID does not exist"
Sep 18 22:51:10.578434 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.578411    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:51:10.584583 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.584560    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-d6c0ac-6667bf46fd-g8tlz"]
Sep 18 22:51:10.599092 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.599068    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:51:10.603771 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:10.603745    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-d6c0ac-6db494dc65-w9frb"]
Sep 18 22:51:12.290990 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:12.290912    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="742d9377-2264-4c69-92d3-b005cf092233" path="/var/lib/kubelet/pods/742d9377-2264-4c69-92d3-b005cf092233/volumes"
Sep 18 22:51:12.291403 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:12.291305    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ee186927-348d-496a-bfe8-9b67a48d5ee4" path="/var/lib/kubelet/pods/ee186927-348d-496a-bfe8-9b67a48d5ee4/volumes"
Sep 18 22:51:13.020042 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:13.020005    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:51:17.357967 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.357929    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"]
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358312    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c5aa114e-3bd1-4473-8391-31a774c2bf20" containerName="revoke-keys"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358324    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c5aa114e-3bd1-4473-8391-31a774c2bf20" containerName="revoke-keys"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358333    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ee186927-348d-496a-bfe8-9b67a48d5ee4" containerName="payload-processing"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358338    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ee186927-348d-496a-bfe8-9b67a48d5ee4" containerName="payload-processing"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358347    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="742d9377-2264-4c69-92d3-b005cf092233" containerName="payload-pre-processing"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358353    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="742d9377-2264-4c69-92d3-b005cf092233" containerName="payload-pre-processing"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358366    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:51:17.358385 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358374    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2fdf193-4f22-4abe-9a85-a6e01eef1c93" containerName="istio-proxy"
Sep 18 22:51:17.358648 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358452    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="742d9377-2264-4c69-92d3-b005cf092233" containerName="payload-pre-processing"
Sep 18 22:51:17.358648 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358461    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c5aa114e-3bd1-4473-8391-31a774c2bf20" containerName="revoke-keys"
Sep 18 22:51:17.358648 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.358468    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ee186927-348d-496a-bfe8-9b67a48d5ee4" containerName="payload-processing"
Sep 18 22:51:17.363329 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.363308    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:17.366630 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.366609    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-8dgjh\""
Sep 18 22:51:17.366767 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.366609    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:51:17.391443 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.391418    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"]
Sep 18 22:51:17.514496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.514464    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk\" (UID: \"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:17.615254 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.615165    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk\" (UID: \"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:17.615755 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.615737    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk\" (UID: \"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:17.673056 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.673002    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:17.949080 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:17.949055    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"]
Sep 18 22:51:17.951785 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:51:17.951755    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9e8a2a65_dc44_4ed9_9edd_1d7eae2a6b9e.slice/crio-5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756 WatchSource:0}: Error finding container 5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756: Status 404 returned error can't find the container with id 5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756
Sep 18 22:51:18.606768 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:18.606679    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk" event={"ID":"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e","Type":"ContainerStarted","Data":"2c2829cd752ba2d1f177818e59cc972901843c8d38541edad73c9cb6132d16e7"}
Sep 18 22:51:18.606768 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:18.606730    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk" event={"ID":"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e","Type":"ContainerStarted","Data":"5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756"}
Sep 18 22:51:18.624012 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:18.623960    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk" podStartSLOduration=1.6239404400000002 podStartE2EDuration="1.62394044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:51:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:51:18.622717907 +0000 UTC m=+2070.804302453" watchObservedRunningTime="2026-09-18 22:51:18.62394044 +0000 UTC m=+2070.805524986"
Sep 18 22:51:19.612654 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:19.612621    2551 generic.go:356] "Generic (PLEG): container finished" podID="9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" containerID="2c2829cd752ba2d1f177818e59cc972901843c8d38541edad73c9cb6132d16e7" exitCode=0
Sep 18 22:51:19.613016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:19.612703    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk" event={"ID":"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e","Type":"ContainerDied","Data":"2c2829cd752ba2d1f177818e59cc972901843c8d38541edad73c9cb6132d16e7"}
Sep 18 22:51:20.755823 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:20.755788    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:20.845010 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:20.844971    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca\") pod \"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e\" (UID: \"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e\") "
Sep 18 22:51:20.845336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:20.845315    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" (UID: "9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:20.945756 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:20.945720    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e-maas-api-service-ca\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:21.624223 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:21.624181    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk" event={"ID":"9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e","Type":"ContainerDied","Data":"5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756"}
Sep 18 22:51:21.624223 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:21.624206    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"
Sep 18 22:51:21.624475 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:21.624237    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c5b52640e3e038fca0165ddafc72416a7e144150fbd419612638f8c78d9c756"
Sep 18 22:51:27.847127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:27.847092    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"]
Sep 18 22:51:27.850160 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:27.850133    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w3-a91c21-29829525-h2565"]
Sep 18 22:51:28.073925 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:28.073893    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"]
Sep 18 22:51:28.077251 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:28.077230    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-a91c21-6b9nk"]
Sep 18 22:51:28.290406 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:28.290375    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" path="/var/lib/kubelet/pods/9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e/volumes"
Sep 18 22:51:28.290665 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:28.290652    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="dff96a01-2731-426a-b32c-1e222614194d" path="/var/lib/kubelet/pods/dff96a01-2731-426a-b32c-1e222614194d/volumes"
Sep 18 22:51:30.356439 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:30.356400    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:51:34.860496 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:34.860463    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:51:34.860869 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:34.860728    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy" containerID="cri-o://c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6" gracePeriod=30
Sep 18 22:51:39.213724 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:39.213691    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:51:40.109203 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.109181    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:51:40.207100 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207076    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207105    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207138    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207162    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207231 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207208    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207242    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207275    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dzdwq\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207344    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207389    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket\") pod \"13d7000e-aef2-474d-bbee-cce25facbae7\" (UID: \"13d7000e-aef2-474d-bbee-cce25facbae7\") "
Sep 18 22:51:40.207463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207447    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs" (OuterVolumeSpecName: "workload-certs") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:40.207731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207506    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data" (OuterVolumeSpecName: "istio-data") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:40.207731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207577    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert" (OuterVolumeSpecName: "istiod-ca-cert") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 18 22:51:40.207731 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207662    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket" (OuterVolumeSpecName: "workload-socket") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:40.207893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207756    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket" (OuterVolumeSpecName: "credential-socket") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:40.207893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207831    2551 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.207893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207851    2551 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-workload-certs\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.207893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207864    2551 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/13d7000e-aef2-474d-bbee-cce25facbae7-istiod-ca-cert\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.207893 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.207876    2551 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-data\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.209400 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.209376    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token" (OuterVolumeSpecName: "istio-token") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:40.209494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.209400    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq" (OuterVolumeSpecName: "kube-api-access-dzdwq") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "kube-api-access-dzdwq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:51:40.209494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.209410    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy" (OuterVolumeSpecName: "istio-envoy") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:51:40.209494 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.209453    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo" (OuterVolumeSpecName: "istio-podinfo") pod "13d7000e-aef2-474d-bbee-cce25facbae7" (UID: "13d7000e-aef2-474d-bbee-cce25facbae7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Sep 18 22:51:40.308690 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.308670    2551 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-istio-envoy\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.308969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.308692    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dzdwq\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-kube-api-access-dzdwq\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.308969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.308702    2551 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/13d7000e-aef2-474d-bbee-cce25facbae7-credential-socket\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.308969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.308710    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/13d7000e-aef2-474d-bbee-cce25facbae7-istio-token\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.308969 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.308720    2551 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/13d7000e-aef2-474d-bbee-cce25facbae7-istio-podinfo\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:51:40.728309 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.728279    2551 generic.go:356] "Generic (PLEG): container finished" podID="13d7000e-aef2-474d-bbee-cce25facbae7" containerID="c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6" exitCode=0
Sep 18 22:51:40.728449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.728329    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"
Sep 18 22:51:40.728449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.728353    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerDied","Data":"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"}
Sep 18 22:51:40.728449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.728395    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh" event={"ID":"13d7000e-aef2-474d-bbee-cce25facbae7","Type":"ContainerDied","Data":"24455a421e656064c4bb0777e0caeac8b340ee153d773211c40a879ec233b370"}
Sep 18 22:51:40.728449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.728419    2551 scope.go:117] "RemoveContainer" containerID="c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"
Sep 18 22:51:40.737339 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.737317    2551 scope.go:117] "RemoveContainer" containerID="b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"
Sep 18 22:51:40.747991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.747968    2551 scope.go:117] "RemoveContainer" containerID="c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"
Sep 18 22:51:40.748359 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:51:40.748333    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6\": container with ID starting with c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6 not found: ID does not exist" containerID="c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"
Sep 18 22:51:40.748788 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.748368    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6"} err="failed to get container status \"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6\": rpc error: code = NotFound desc = could not find container \"c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6\": container with ID starting with c5876c03d1e960c00c2c07ffff7968d8737eb6d2ca8f718df5d9fa5ede35fbf6 not found: ID does not exist"
Sep 18 22:51:40.748788 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.748749    2551 scope.go:117] "RemoveContainer" containerID="b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"
Sep 18 22:51:40.751009 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:51:40.749443    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d\": container with ID starting with b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d not found: ID does not exist" containerID="b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"
Sep 18 22:51:40.751009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.749497    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d"} err="failed to get container status \"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d\": rpc error: code = NotFound desc = could not find container \"b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d\": container with ID starting with b2f0ca020776aa08296307ad7f394fbe2d3b7557dc5e4272b3730ac07d2f038d not found: ID does not exist"
Sep 18 22:51:40.751009 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.750958    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:51:40.756076 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:40.756049    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w3-a91c21-openshift-default-6b7b44d48-j2jnh"]
Sep 18 22:51:42.290846 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:42.290813    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" path="/var/lib/kubelet/pods/13d7000e-aef2-474d-bbee-cce25facbae7/volumes"
Sep 18 22:51:48.421183 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:48.421157    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:51:48.424802 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:48.424774    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:51:48.820556 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:51:48.820485    2551 scope.go:117] "RemoveContainer" containerID="f327fc03c055d6f7ab89e20c0b7bd31550aeba3f75569b236a84db49db04fdbf"
Sep 18 22:53:01.985242 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:53:01.985204    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:53:41.889944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:53:41.889909    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:53:48.869458 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:53:48.869429    2551 scope.go:117] "RemoveContainer" containerID="865d50ec0ef10b0968eb4a6254ca2ab7114d2e672e498a9e22d42ae57d989014"
Sep 18 22:53:49.885074 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:53:49.885024    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:53:57.695952 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:53:57.695915    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:54:15.902652 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:15.902571    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:54:21.599147 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:21.599115    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:54:23.308843 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:23.308813    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:54:23.309303 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:23.309001    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager" containerID="cri-o://6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a" gracePeriod=10
Sep 18 22:54:25.353572 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.353541    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:54:25.426351 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.426325    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume\") pod \"18261b97-897d-4a5d-b471-75099643a20f\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") "
Sep 18 22:54:25.426513 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.426373    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qbmpc\" (UniqueName: \"kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc\") pod \"18261b97-897d-4a5d-b471-75099643a20f\" (UID: \"18261b97-897d-4a5d-b471-75099643a20f\") "
Sep 18 22:54:25.426710 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.426686    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "18261b97-897d-4a5d-b471-75099643a20f" (UID: "18261b97-897d-4a5d-b471-75099643a20f"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 18 22:54:25.428622 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.428591    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc" (OuterVolumeSpecName: "kube-api-access-qbmpc") pod "18261b97-897d-4a5d-b471-75099643a20f" (UID: "18261b97-897d-4a5d-b471-75099643a20f"). InnerVolumeSpecName "kube-api-access-qbmpc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 18 22:54:25.527271 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.527247    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/18261b97-897d-4a5d-b471-75099643a20f-extensions-socket-volume\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:54:25.527271 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.527269    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbmpc\" (UniqueName: \"kubernetes.io/projected/18261b97-897d-4a5d-b471-75099643a20f-kube-api-access-qbmpc\") on node \"ip-10-0-129-37.ec2.internal\" DevicePath \"\""
Sep 18 22:54:25.575127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.575100    2551 generic.go:356] "Generic (PLEG): container finished" podID="18261b97-897d-4a5d-b471-75099643a20f" containerID="6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a" exitCode=0
Sep 18 22:54:25.575211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.575151    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"
Sep 18 22:54:25.575211 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.575179    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerDied","Data":"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"}
Sep 18 22:54:25.575283 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.575231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6" event={"ID":"18261b97-897d-4a5d-b471-75099643a20f","Type":"ContainerDied","Data":"45718b8e6e8bd7471da074683114973abf9576d1975fede3646d2029fca06d30"}
Sep 18 22:54:25.575283 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.575254    2551 scope.go:117] "RemoveContainer" containerID="6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"
Sep 18 22:54:25.584336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.584318    2551 scope.go:117] "RemoveContainer" containerID="3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"
Sep 18 22:54:25.592364 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.592347    2551 scope.go:117] "RemoveContainer" containerID="6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"
Sep 18 22:54:25.592646 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:54:25.592627    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a\": container with ID starting with 6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a not found: ID does not exist" containerID="6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"
Sep 18 22:54:25.592746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.592653    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a"} err="failed to get container status \"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a\": rpc error: code = NotFound desc = could not find container \"6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a\": container with ID starting with 6352e97409a2afc36d6ac35d385aef467be5bc2f990e3b02f7848f60cf77090a not found: ID does not exist"
Sep 18 22:54:25.592746 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.592669    2551 scope.go:117] "RemoveContainer" containerID="3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"
Sep 18 22:54:25.592934 ip-10-0-129-37 kubenswrapper[2551]: E0918 22:54:25.592912    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe\": container with ID starting with 3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe not found: ID does not exist" containerID="3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"
Sep 18 22:54:25.592989 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.592946    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe"} err="failed to get container status \"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe\": rpc error: code = NotFound desc = could not find container \"3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe\": container with ID starting with 3a0e6b8d021829a910d49c0ab4dc7014d9adcddd7182636491804e8844d73efe not found: ID does not exist"
Sep 18 22:54:25.601958 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.601937    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:54:25.608122 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:25.608097    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9wrj6"]
Sep 18 22:54:26.290519 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:54:26.290491    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="18261b97-897d-4a5d-b471-75099643a20f" path="/var/lib/kubelet/pods/18261b97-897d-4a5d-b471-75099643a20f/volumes"
Sep 18 22:55:24.048591 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.048551    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"]
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049059    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049080    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049112    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" containerName="revoke-keys"
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049120    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" containerName="revoke-keys"
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049135    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:55:24.049215 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049144    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:55:24.049539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049239    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9e8a2a65-dc44-4ed9-9edd-1d7eae2a6b9e" containerName="revoke-keys"
Sep 18 22:55:24.049539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049251    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:55:24.049539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049260    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:55:24.049539 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.049270    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:55:24.051973 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.051952    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.054472 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.054455    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-v2hc2\""
Sep 18 22:55:24.067183 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.067160    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"]
Sep 18 22:55:24.172817 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.172785    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6cb7\" (UniqueName: \"kubernetes.io/projected/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-kube-api-access-q6cb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.172904 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.172847    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.273347 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.273327    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.273451 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.273392    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6cb7\" (UniqueName: \"kubernetes.io/projected/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-kube-api-access-q6cb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.273675 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.273655    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.284999 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.284975    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6cb7\" (UniqueName: \"kubernetes.io/projected/9f830be7-1a0d-4f1c-a16c-52b03ea519b0-kube-api-access-q6cb7\") pod \"kuadrant-operator-controller-manager-5769f99d9f-7jl57\" (UID: \"9f830be7-1a0d-4f1c-a16c-52b03ea519b0\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.361852 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.361799    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.634591 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.634570    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"]
Sep 18 22:55:24.636856 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:55:24.636828    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f830be7_1a0d_4f1c_a16c_52b03ea519b0.slice/crio-e06ea147f4903b078ef397df738988700e72a52362688d4a2cf034fee7b30543 WatchSource:0}: Error finding container e06ea147f4903b078ef397df738988700e72a52362688d4a2cf034fee7b30543: Status 404 returned error can't find the container with id e06ea147f4903b078ef397df738988700e72a52362688d4a2cf034fee7b30543
Sep 18 22:55:24.866728 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.866694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57" event={"ID":"9f830be7-1a0d-4f1c-a16c-52b03ea519b0","Type":"ContainerStarted","Data":"be334d1328d0575fdfbd4eb9bb8808fb742093ad79e964c9102d18850f2e8ce7"}
Sep 18 22:55:24.866872 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.866749    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57" event={"ID":"9f830be7-1a0d-4f1c-a16c-52b03ea519b0","Type":"ContainerStarted","Data":"e06ea147f4903b078ef397df738988700e72a52362688d4a2cf034fee7b30543"}
Sep 18 22:55:24.866872 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.866776    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:24.888415 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:24.888341    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57" podStartSLOduration=0.888327093 podStartE2EDuration="888.327093ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:55:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:55:24.887276053 +0000 UTC m=+2317.068860598" watchObservedRunningTime="2026-09-18 22:55:24.888327093 +0000 UTC m=+2317.069911638"
Sep 18 22:55:35.874839 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:35.874806    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-7jl57"
Sep 18 22:55:48.905330 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:48.905251    2551 scope.go:117] "RemoveContainer" containerID="07b1c9bfda875304e113c540b7a04b2ba566e2f21ca12000902fdb9299e45270"
Sep 18 22:55:48.914392 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:48.914370    2551 scope.go:117] "RemoveContainer" containerID="b56a2e703f54b49bb893276d9ead786f46a3c6039dcd2ed5e3a77af7d9265d43"
Sep 18 22:55:51.509281 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:51.509245    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:55:57.392574 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:55:57.392539    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:56:07.898130 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:07.898089    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:56:20.792495 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:20.792462    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:56:41.597689 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:41.597657    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:56:48.452599 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:48.452549    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:56:48.458333 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:48.458312    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ovn-kubernetes_ovnkube-node-g7g7q_fcd4e641-bee5-4d87-b8bb-80e070cf7461/ovn-acl-logging/0.log"
Sep 18 22:56:48.937410 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:48.937376    2551 scope.go:117] "RemoveContainer" containerID="2bcbc8428764e68549e9630c536d54b2c77bf60748b5eaa59e48b0ffa2d6c0ec"
Sep 18 22:56:57.793867 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:56:57.793832    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:57:05.498336 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.498301    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:57:05.805071 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.804985    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"]
Sep 18 22:57:05.805340 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.805328    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:57:05.805384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.805341    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="18261b97-897d-4a5d-b471-75099643a20f" containerName="manager"
Sep 18 22:57:05.805384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.805361    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:57:05.805384 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.805367    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:57:05.805483 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.805420    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="13d7000e-aef2-474d-bbee-cce25facbae7" containerName="istio-proxy"
Sep 18 22:57:05.808315 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.808292    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:05.811529 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.811509    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Sep 18 22:57:05.811628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.811545    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Sep 18 22:57:05.811628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.811582    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-7xxrx\""
Sep 18 22:57:05.811628 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.811613    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Sep 18 22:57:05.811862 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.811847    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 18 22:57:05.818635 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.818612    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"]
Sep 18 22:57:05.963723 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.963701    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-openshift-service-ca\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:05.963847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.963732    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-97z6p\" (UniqueName: \"kubernetes.io/projected/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-kube-api-access-97z6p\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:05.963847 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.963805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-metrics-certs\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:05.963929 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:05.963850    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-maas-api-tls\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.064887 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.064824    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-metrics-certs\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.064887 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.064867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-maas-api-tls\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.065077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.064887    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-openshift-service-ca\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.065077 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.064905    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-97z6p\" (UniqueName: \"kubernetes.io/projected/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-kube-api-access-97z6p\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.065479 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.065457    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-openshift-service-ca\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.067290 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.067272    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-maas-api-tls\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.067380 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.067355    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-metrics-certs\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.072200 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.072181    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-97z6p\" (UniqueName: \"kubernetes.io/projected/ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8-kube-api-access-97z6p\") pod \"maas-api-68c86dfd56-8nw5j\" (UID: \"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8\") " pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.120188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.120165    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:06.390588 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.390559    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"]
Sep 18 22:57:06.392973 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:57:06.392946    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podef681e0c_cd7e_48ff_b22a_f2656a3b2bb8.slice/crio-8592d22a83425c1896a47f72ad54ee18522e017f5e4203398666a6b73cfb864b WatchSource:0}: Error finding container 8592d22a83425c1896a47f72ad54ee18522e017f5e4203398666a6b73cfb864b: Status 404 returned error can't find the container with id 8592d22a83425c1896a47f72ad54ee18522e017f5e4203398666a6b73cfb864b
Sep 18 22:57:06.394280 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:06.394260    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 18 22:57:07.379971 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:07.379941    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j" event={"ID":"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8","Type":"ContainerStarted","Data":"fd6859df6abf89aaac8580ce75426039228f59e7f60837d1679c33f6c69845ca"}
Sep 18 22:57:07.380376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:07.379986    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j" event={"ID":"ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8","Type":"ContainerStarted","Data":"8592d22a83425c1896a47f72ad54ee18522e017f5e4203398666a6b73cfb864b"}
Sep 18 22:57:07.380376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:07.380081    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:07.397957 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:07.397913    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j" podStartSLOduration=2.028710413 podStartE2EDuration="2.39790011s" totalImagesPullingTime="369.189697ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:57:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:57:07.396962947 +0000 UTC m=+2419.578547492" watchObservedRunningTime="2026-09-18 22:57:07.39790011 +0000 UTC m=+2419.579484655"
Sep 18 22:57:12.092348 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:12.092263    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:57:13.390759 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:13.390729    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-68c86dfd56-8nw5j"
Sep 18 22:57:17.802828 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:17.802794    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:57:24.485887 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:24.485846    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-84994f7ffc-6sllq"]
Sep 18 22:57:33.824737 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:33.824707    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65c85dbc8b-xc6nz_01c8de87-3bd7-4f76-88aa-cd5126e73cb6/copy-manifests/0.log"
Sep 18 22:57:33.843449 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:33.843424    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65c85dbc8b-xc6nz_01c8de87-3bd7-4f76-88aa-cd5126e73cb6/manager/0.log"
Sep 18 22:57:34.291991 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:34.291964    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-68c86dfd56-8nw5j_ef681e0c-cd7e-48ff-b22a-f2656a3b2bb8/maas-api/0.log"
Sep 18 22:57:34.519264 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:34.519238    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829510-hkklr_37ed21b9-d433-4923-a913-db5ed5844367/cleanup/0.log"
Sep 18 22:57:34.625295 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:34.625227    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-key-cleanup-29829525-6gjr5_5e023a35-19da-4261-b88e-7202e6d64d93/cleanup/0.log"
Sep 18 22:57:35.714890 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:35.714852    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rjkmb_28893176-b880-4223-addf-fdced418f439/manager/0.log"
Sep 18 22:57:35.948821 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:35.948794    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7jl57_9f830be7-1a0d-4f1c-a16c-52b03ea519b0/manager/0.log"
Sep 18 22:57:36.061376 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:36.061313    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6sllq_cc33c016-d525-4979-bfb3-9f226deb133d/limitador/0.log"
Sep 18 22:57:36.750871 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:36.750836    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b786b9564-bsppv_66432609-045a-46a7-84f8-3bbef484f2cc/kube-auth-proxy/0.log"
Sep 18 22:57:37.985902 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:37.985873    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-598466548d-sh8gb_0c00fc26-966d-4397-8709-83f94c93f7d2/main/0.log"
Sep 18 22:57:38.326629 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:38.326554    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_premium-simulated-simulated-premium-kserve-57fc66f697-v2wd5_a0e99161-4fe4-45a0-992f-d4dc0b1d9d7e/main/0.log"
Sep 18 22:57:44.092603 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.092558    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-558dg/must-gather-lqs9q"]
Sep 18 22:57:44.096176 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.096155    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.099141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.099109    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-558dg\"/\"default-dockercfg-nqbcn\""
Sep 18 22:57:44.099141 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.099109    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-558dg\"/\"openshift-service-ca.crt\""
Sep 18 22:57:44.099317 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.099156    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-558dg\"/\"kube-root-ca.crt\""
Sep 18 22:57:44.117156 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.117131    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-558dg/must-gather-lqs9q"]
Sep 18 22:57:44.137887 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.137862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-must-gather-output\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.137992 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.137900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tln5\" (UniqueName: \"kubernetes.io/projected/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-kube-api-access-8tln5\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.239127 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.239102    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-must-gather-output\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.239230 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.239136    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tln5\" (UniqueName: \"kubernetes.io/projected/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-kube-api-access-8tln5\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.239417 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.239396    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"must-gather-output\" (UniqueName: \"kubernetes.io/empty-dir/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-must-gather-output\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.255107 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.255084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tln5\" (UniqueName: \"kubernetes.io/projected/6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9-kube-api-access-8tln5\") pod \"must-gather-lqs9q\" (UID: \"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9\") " pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.406041 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.406005    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-558dg/must-gather-lqs9q"
Sep 18 22:57:44.679083 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:44.679057    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-558dg/must-gather-lqs9q"]
Sep 18 22:57:44.680892 ip-10-0-129-37 kubenswrapper[2551]: W0918 22:57:44.680851    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6c9d5ddf_e51b_4627_bd9a_dc44cfef16b9.slice/crio-93cb4bf0384f14b252e8f7514ae9fc5dbea2e1cfbc69180bf5e73966a1923591 WatchSource:0}: Error finding container 93cb4bf0384f14b252e8f7514ae9fc5dbea2e1cfbc69180bf5e73966a1923591: Status 404 returned error can't find the container with id 93cb4bf0384f14b252e8f7514ae9fc5dbea2e1cfbc69180bf5e73966a1923591
Sep 18 22:57:45.572582 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:45.572544    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-558dg/must-gather-lqs9q" event={"ID":"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9","Type":"ContainerStarted","Data":"d34a19ef948c270af1d16210b402c6bf8d180ec8a056b493a51ee23a29446825"}
Sep 18 22:57:45.572926 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:45.572602    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-558dg/must-gather-lqs9q" event={"ID":"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9","Type":"ContainerStarted","Data":"93cb4bf0384f14b252e8f7514ae9fc5dbea2e1cfbc69180bf5e73966a1923591"}
Sep 18 22:57:46.583573 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:46.583526    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-558dg/must-gather-lqs9q" event={"ID":"6c9d5ddf-e51b-4627-bd9a-dc44cfef16b9","Type":"ContainerStarted","Data":"09ae4c88d8f8308ce734fa3bb8b99dc9babb3115e7b5406b695b7914537867c8"}
Sep 18 22:57:46.602954 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:46.602898    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-558dg/must-gather-lqs9q" podStartSLOduration=1.88994063 podStartE2EDuration="2.602876279s" totalImagesPullingTime="712.935649ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-18 22:57:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:57:46.602153335 +0000 UTC m=+2458.783737882" watchObservedRunningTime="2026-09-18 22:57:46.602876279 +0000 UTC m=+2458.784460825"
Sep 18 22:57:47.290554 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:47.290523    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-w6mzb_881581b3-32aa-456a-8c0f-fa4b9a4f8c6b/global-pull-secret-syncer/0.log"
Sep 18 22:57:47.428994 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:47.428956    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-pjg4b_30939455-de05-41c9-8edb-a4b157b28534/konnectivity-agent/0.log"
Sep 18 22:57:47.465071 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:47.465022    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-129-37.ec2.internal_b2d86a8c0ee26cd8a3423d7b77f0c8af/haproxy/0.log"
Sep 18 22:57:48.958936 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:48.958901    2551 scope.go:117] "RemoveContainer" containerID="2c2829cd752ba2d1f177818e59cc972901843c8d38541edad73c9cb6132d16e7"
Sep 18 22:57:48.979673 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:48.979648    2551 scope.go:117] "RemoveContainer" containerID="256ecc4bcc0304f5452e0a83cb621977c52594f3660b20f8cc2eed2e86d97e2f"
Sep 18 22:57:52.308438 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:52.308341    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-rjkmb_28893176-b880-4223-addf-fdced418f439/manager/0.log"
Sep 18 22:57:52.386188 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:52.386147    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-7jl57_9f830be7-1a0d-4f1c-a16c-52b03ea519b0/manager/0.log"
Sep 18 22:57:52.413014 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:52.412986    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-limitador-84994f7ffc-6sllq_cc33c016-d525-4979-bfb3-9f226deb133d/limitador/0.log"
Sep 18 22:57:53.843184 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:53.843158    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/alertmanager/0.log"
Sep 18 22:57:53.874534 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:53.874504    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/config-reloader/0.log"
Sep 18 22:57:53.946830 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:53.946796    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/kube-rbac-proxy-web/0.log"
Sep 18 22:57:53.994794 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:53.994710    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/kube-rbac-proxy/0.log"
Sep 18 22:57:54.053693 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.053658    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/kube-rbac-proxy-metric/0.log"
Sep 18 22:57:54.098050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.098014    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/prom-label-proxy/0.log"
Sep 18 22:57:54.141983 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.141957    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_alertmanager-main-0_983b7fc0-60e4-4ca0-8980-4526551a0cc0/init-config-reloader/0.log"
Sep 18 22:57:54.427614 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.427586    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9tq6_602ce902-1acc-4181-8da1-93fff9d8c6e5/node-exporter/0.log"
Sep 18 22:57:54.463235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.463199    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9tq6_602ce902-1acc-4181-8da1-93fff9d8c6e5/kube-rbac-proxy/0.log"
Sep 18 22:57:54.502766 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.502734    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-d9tq6_602ce902-1acc-4181-8da1-93fff9d8c6e5/init-textfile/0.log"
Sep 18 22:57:54.707416 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.707390    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qx8x7_dfb09f48-c756-4259-86ff-2395ef6b9c06/kube-rbac-proxy-main/0.log"
Sep 18 22:57:54.740482 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.740457    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qx8x7_dfb09f48-c756-4259-86ff-2395ef6b9c06/kube-rbac-proxy-self/0.log"
Sep 18 22:57:54.776302 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.776270    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-86c5bc79d6-qx8x7_dfb09f48-c756-4259-86ff-2395ef6b9c06/openshift-state-metrics/0.log"
Sep 18 22:57:54.832946 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.832912    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/prometheus/0.log"
Sep 18 22:57:54.858637 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.858601    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/config-reloader/0.log"
Sep 18 22:57:54.888102 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.888078    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/thanos-sidecar/0.log"
Sep 18 22:57:54.916845 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.916809    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/kube-rbac-proxy-web/0.log"
Sep 18 22:57:54.943517 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.943470    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/kube-rbac-proxy/0.log"
Sep 18 22:57:54.974297 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:54.974225    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/kube-rbac-proxy-thanos/0.log"
Sep 18 22:57:55.000381 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:55.000357    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_a06ca16d-bb05-43dd-ae67-c7f6e40f1cf6/init-config-reloader/0.log"
Sep 18 22:57:55.046919 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:55.046891    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4qn5j_3a39227a-ff22-4ef1-b877-2f2fa394474b/prometheus-operator/0.log"
Sep 18 22:57:55.080484 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:55.080451    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-7bd5d56d6-4qn5j_3a39227a-ff22-4ef1-b877-2f2fa394474b/kube-rbac-proxy/0.log"
Sep 18 22:57:56.640736 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:56.640700    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-network-console_networking-console-plugin-6446c8d75d-jmzd9_5be494a2-c1ec-497f-b8ab-42a9d0b9529b/networking-console-plugin/0.log"
Sep 18 22:57:56.989131 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:56.989092    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"]
Sep 18 22:57:56.996873 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:56.996842    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.000506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.000472    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"]
Sep 18 22:57:57.072235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.071843    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-lib-modules\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.072235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.071902    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-proc\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.072235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.071950    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-podres\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.072235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.072050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kwtwv\" (UniqueName: \"kubernetes.io/projected/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-kube-api-access-kwtwv\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.072235 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.072107    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-sys\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.172982 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.172950    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kwtwv\" (UniqueName: \"kubernetes.io/projected/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-kube-api-access-kwtwv\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.172997    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-sys\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173045    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-lib-modules\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173072    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-proc\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-podres\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173118    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-sys\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173186 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173183    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-proc\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173234    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-podres\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.173454 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.173284    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-lib-modules\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.182501 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.182448    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kwtwv\" (UniqueName: \"kubernetes.io/projected/5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7-kube-api-access-kwtwv\") pod \"perf-node-gather-daemonset-4l4hc\" (UID: \"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7\") " pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.310909 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.310815    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:57.629421 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.626992    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"]
Sep 18 22:57:57.670050 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:57.669401    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc" event={"ID":"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7","Type":"ContainerStarted","Data":"ed2294a5f6796c8666461f577a85fcd441ab7373450a4c59b6b9ab4362c653f5"}
Sep 18 22:57:58.022944 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:58.022911    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-console_downloads-6594f4c75-v9jlt_1435f7fe-3c57-4700-ac27-f9fac9f5d622/download-server/0.log"
Sep 18 22:57:58.678168 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:58.678134    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc" event={"ID":"5e0bf7d2-a8b6-4bbc-bbd0-20e687f045c7","Type":"ContainerStarted","Data":"4599a3339a4a7ba554933578e3af5ce53db68ab29a4fa322b5c432b1795421cd"}
Sep 18 22:57:58.678707 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:58.678685    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:57:58.700706 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:57:58.700651    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc" podStartSLOduration=2.700636222 podStartE2EDuration="2.700636222s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-18 22:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-18 22:57:58.697408376 +0000 UTC m=+2470.878992924" watchObservedRunningTime="2026-09-18 22:57:58.700636222 +0000 UTC m=+2470.882220810"
Sep 18 22:58:00.210146 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:00.210112    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-txlsf_fcd712ed-b052-43bd-a7b2-691fcb170e5e/dns/0.log"
Sep 18 22:58:00.276848 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:00.276798    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-txlsf_fcd712ed-b052-43bd-a7b2-691fcb170e5e/kube-rbac-proxy/0.log"
Sep 18 22:58:00.406800 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:00.406771    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-v7kdp_ff71cfa3-3462-41c0-93fe-6118322980e0/dns-node-resolver/0.log"
Sep 18 22:58:01.085979 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:01.085952    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-mbrwj_c0777467-72d9-4cb1-89de-35f82dc26ec2/node-ca/0.log"
Sep 18 22:58:02.617323 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:02.617294    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b786b9564-bsppv_66432609-045a-46a7-84f8-3bbef484f2cc/kube-auth-proxy/0.log"
Sep 18 22:58:03.773868 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:03.773832    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9qkp4_57160a47-a996-4742-b9e3-7a3d9cd996e9/serve-healthcheck-canary/0.log"
Sep 18 22:58:04.275615 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:04.275583    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tsqh_43098664-d625-4793-8ac6-dfadfead8f26/kube-rbac-proxy/0.log"
Sep 18 22:58:04.298715 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:04.298681    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tsqh_43098664-d625-4793-8ac6-dfadfead8f26/exporter/0.log"
Sep 18 22:58:04.321733 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:04.321708    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-4tsqh_43098664-d625-4793-8ac6-dfadfead8f26/extractor/0.log"
Sep 18 22:58:05.701967 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:05.701929    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-558dg/perf-node-gather-daemonset-4l4hc"
Sep 18 22:58:07.118259 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:07.118166    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65c85dbc8b-xc6nz_01c8de87-3bd7-4f76-88aa-cd5126e73cb6/manager/0.log"
Sep 18 22:58:07.143506 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:07.143480    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-65c85dbc8b-xc6nz_01c8de87-3bd7-4f76-88aa-cd5126e73cb6/copy-manifests/0.log"
Sep 18 22:58:07.438319 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:07.438287    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b786b9564-bsppv_66432609-045a-46a7-84f8-3bbef484f2cc/kube-auth-proxy/0.log"
Sep 18 22:58:08.623445 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:08.623365    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_lws-controller-manager-5bdd6569c9-d7hp2_672eccbc-27de-424e-b198-09a14f3d4538/manager/0.log"
Sep 18 22:58:08.654016 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:08.653970    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-lws-operator_openshift-lws-operator-fd8ccff4c-7n77f_88e16948-5195-4cab-88ae-6a129e4a3214/openshift-lws-operator/0.log"
Sep 18 22:58:09.623463 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:09.623436    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-6b786b9564-bsppv_66432609-045a-46a7-84f8-3bbef484f2cc/kube-auth-proxy/0.log"
Sep 18 22:58:13.507734 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:13.507701    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-52kqb_92223b3f-6257-4aea-aad6-283fc83005bf/migrator/0.log"
Sep 18 22:58:13.533525 ip-10-0-129-37 kubenswrapper[2551]: I0918 22:58:13.533500    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-76794cbd6c-52kqb_92223b3f-6257-4aea-aad6-283fc83005bf/graceful-termination/0.log"