Sep 11 13:17:30.998144 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Sep 11 13:17:30.998154 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Sep 11 13:17:30.998161 ip-10-0-141-50 systemd[1]: kubelet.service: Failed with result 'resources'.
Sep 11 13:17:30.998403 ip-10-0-141-50 systemd[1]: Failed to start Kubernetes Kubelet.
Sep 11 13:17:41.494456 ip-10-0-141-50 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Sep 11 13:17:41.494467 ip-10-0-141-50 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot 9a72cb6c8b90401ba494e9af1a7640f3 --
Sep 11 13:19:28.372926 ip-10-0-141-50 systemd[1]: Starting Kubernetes Kubelet...
Sep 11 13:19:28.802867 ip-10-0-141-50 kubenswrapper[2551]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:19:28.802867 ip-10-0-141-50 kubenswrapper[2551]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Sep 11 13:19:28.802867 ip-10-0-141-50 kubenswrapper[2551]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:19:28.802867 ip-10-0-141-50 kubenswrapper[2551]: Flag --pod-infra-container-image has been deprecated, will be removed in 1.35. Image garbage collector will get sandbox image information from CRI.
Sep 11 13:19:28.802867 ip-10-0-141-50 kubenswrapper[2551]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Sep 11 13:19:28.803521 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.802903    2551 server.go:213] "--pod-infra-container-image will not be pruned by the image garbage collector in kubelet and should also be set in the remote runtime"
Sep 11 13:19:28.806902 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806885    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.806902 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806901    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.806902 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806904    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806909    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806912    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806914    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806917    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806920    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806923    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806926    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806928    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806932    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806935    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806937    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806941    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806944    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806946    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806949    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806952    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806955    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806957    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806965    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.806992 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806968    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806970    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806973    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806976    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806980    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806984    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806987    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806990    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806993    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806996    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.806999    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807002    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807005    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807008    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807011    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807013    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807016    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807018    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807021    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.807464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807023    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807026    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807029    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807031    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807034    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807036    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807039    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807043    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807046    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807048    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807051    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807053    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807056    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807059    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807061    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807066    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807069    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807071    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807074    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807076    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.807997 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807079    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807083    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807086    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807089    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807092    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807095    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807098    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807101    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807104    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807107    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807110    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807113    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807116    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807119    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807122    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807124    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807127    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807130    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807132    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.808786 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807135    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807137    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807140    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807143    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807145    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807148    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807151    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807153    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807156    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807159    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807163    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807166    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807168    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807171    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807174    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807176    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807179    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807183    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.807187    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808201    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.809278 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808216    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808222    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808233    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808238    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808242    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808247    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808253    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808259    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808264    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808268    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808274    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808279    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808283    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808288    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808297    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808302    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808307    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808311    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808315    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.809778 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808320    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808326    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808331    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808335    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808348    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808353    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808357    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808361    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808370    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808374    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808380    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808386    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808391    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808395    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808400    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808404    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808408    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808413    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808417    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.810244 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808428    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808432    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808437    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808441    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808446    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808450    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808454    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808458    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808463    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808467    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808471    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808475    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808480    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808488    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808495    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808500    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808505    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808509    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808513    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808518    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.810765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808522    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808526    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808531    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808535    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808539    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808544    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808554    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808558    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808562    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808567    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808572    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808578    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808586    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808591    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808595    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808600    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808604    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808613    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808617    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.811256 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808622    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808641    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808646    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808650    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808654    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808658    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808663    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808668    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808673    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808678    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808682    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808691    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808695    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808699    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808704    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808708    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808713    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808717    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808722    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808726    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.811736 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.808730    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809422    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809434    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809438    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809441    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809445    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809448    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809451    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809454    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809457    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809460    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809462    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809465    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809468    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809472    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809476    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809479    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809482    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809484    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.812237 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809487    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809490    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809493    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809496    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809499    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809501    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809504    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809506    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809509    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809511    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809515    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809517    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809520    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809522    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809525    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809528    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809531    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809534    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809536    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.812734 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809539    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809542    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809545    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809547    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809550    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809553    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809555    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809558    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809561    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809563    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809566    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809568    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809571    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809574    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809576    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809579    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809581    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809584    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809586    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809589    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.813209 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809592    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809594    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809597    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809599    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809602    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809604    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809608    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809616    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809619    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809622    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809639    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809643    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809646    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809649    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809652    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809654    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809657    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809659    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809662    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.813780 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809665    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809672    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809674    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809677    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809679    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809682    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809691    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809693    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809696    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809698    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809701    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809704    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809706    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809709    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809711    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809714    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809717    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809719    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809722    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.814251 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809726    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809731    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809735    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.809738    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809813    2551 flags.go:64] FLAG: --address="0.0.0.0"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809850    2551 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809856    2551 flags.go:64] FLAG: --anonymous-auth="true"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809861    2551 flags.go:64] FLAG: --application-metrics-count-limit="100"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809865    2551 flags.go:64] FLAG: --authentication-token-webhook="false"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809868    2551 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809872    2551 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809876    2551 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809879    2551 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809882    2551 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809886    2551 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809889    2551 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809892    2551 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809895    2551 flags.go:64] FLAG: --cgroup-root=""
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809898    2551 flags.go:64] FLAG: --cgroups-per-qos="true"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809901    2551 flags.go:64] FLAG: --client-ca-file=""
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809912    2551 flags.go:64] FLAG: --cloud-provider="external"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809915    2551 flags.go:64] FLAG: --cluster-dns="[]"
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809920    2551 flags.go:64] FLAG: --cluster-domain=""
Sep 11 13:19:28.814727 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809923    2551 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809926    2551 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809929    2551 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809932    2551 flags.go:64] FLAG: --container-log-max-files="5"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809936    2551 flags.go:64] FLAG: --container-log-max-size="10Mi"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809939    2551 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809942    2551 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809945    2551 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809948    2551 flags.go:64] FLAG: --contention-profiling="false"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809951    2551 flags.go:64] FLAG: --cpu-cfs-quota="true"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809956    2551 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809959    2551 flags.go:64] FLAG: --cpu-manager-policy="none"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809962    2551 flags.go:64] FLAG: --cpu-manager-policy-options=""
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809965    2551 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809968    2551 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809971    2551 flags.go:64] FLAG: --enable-debugging-handlers="true"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809974    2551 flags.go:64] FLAG: --enable-load-reader="false"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809977    2551 flags.go:64] FLAG: --enable-server="true"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809980    2551 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809984    2551 flags.go:64] FLAG: --event-burst="100"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809987    2551 flags.go:64] FLAG: --event-qps="50"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809990    2551 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809993    2551 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809996    2551 flags.go:64] FLAG: --eviction-hard=""
Sep 11 13:19:28.815288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.809999    2551 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810002    2551 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810005    2551 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810008    2551 flags.go:64] FLAG: --eviction-soft=""
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810011    2551 flags.go:64] FLAG: --eviction-soft-grace-period=""
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810014    2551 flags.go:64] FLAG: --exit-on-lock-contention="false"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810020    2551 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810023    2551 flags.go:64] FLAG: --experimental-mounter-path=""
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810026    2551 flags.go:64] FLAG: --fail-cgroupv1="false"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810029    2551 flags.go:64] FLAG: --fail-swap-on="true"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810031    2551 flags.go:64] FLAG: --feature-gates=""
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810035    2551 flags.go:64] FLAG: --file-check-frequency="20s"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810038    2551 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810041    2551 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810044    2551 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810047    2551 flags.go:64] FLAG: --healthz-port="10248"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810050    2551 flags.go:64] FLAG: --help="false"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810053    2551 flags.go:64] FLAG: --hostname-override="ip-10-0-141-50.ec2.internal"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810056    2551 flags.go:64] FLAG: --housekeeping-interval="10s"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810060    2551 flags.go:64] FLAG: --http-check-frequency="20s"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810063    2551 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810066    2551 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810069    2551 flags.go:64] FLAG: --image-gc-high-threshold="85"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810072    2551 flags.go:64] FLAG: --image-gc-low-threshold="80"
Sep 11 13:19:28.815871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810075    2551 flags.go:64] FLAG: --image-service-endpoint=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810078    2551 flags.go:64] FLAG: --kernel-memcg-notification="false"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810081    2551 flags.go:64] FLAG: --kube-api-burst="100"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810084    2551 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810087    2551 flags.go:64] FLAG: --kube-api-qps="50"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810090    2551 flags.go:64] FLAG: --kube-reserved=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810093    2551 flags.go:64] FLAG: --kube-reserved-cgroup=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810096    2551 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810099    2551 flags.go:64] FLAG: --kubelet-cgroups=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810102    2551 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810105    2551 flags.go:64] FLAG: --lock-file=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810108    2551 flags.go:64] FLAG: --log-cadvisor-usage="false"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810111    2551 flags.go:64] FLAG: --log-flush-frequency="5s"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810114    2551 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810119    2551 flags.go:64] FLAG: --log-json-split-stream="false"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810128    2551 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810131    2551 flags.go:64] FLAG: --log-text-split-stream="false"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810133    2551 flags.go:64] FLAG: --logging-format="text"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810136    2551 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810140    2551 flags.go:64] FLAG: --make-iptables-util-chains="true"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810142    2551 flags.go:64] FLAG: --manifest-url=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810146    2551 flags.go:64] FLAG: --manifest-url-header=""
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810150    2551 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810153    2551 flags.go:64] FLAG: --max-open-files="1000000"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810157    2551 flags.go:64] FLAG: --max-pods="110"
Sep 11 13:19:28.816518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810160    2551 flags.go:64] FLAG: --maximum-dead-containers="-1"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810163    2551 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810166    2551 flags.go:64] FLAG: --memory-manager-policy="None"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810169    2551 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810172    2551 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810175    2551 flags.go:64] FLAG: --node-ip="0.0.0.0"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810178    2551 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810196    2551 flags.go:64] FLAG: --node-status-max-images="50"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810200    2551 flags.go:64] FLAG: --node-status-update-frequency="10s"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810204    2551 flags.go:64] FLAG: --oom-score-adj="-999"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810207    2551 flags.go:64] FLAG: --pod-cidr=""
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810210    2551 flags.go:64] FLAG: --pod-infra-container-image="quay.io/openshift-release-dev/ocp-v4.0-art-dev@sha256:eee392dbe3ac2ed4a38bca29ae3bd486f5b7780406c62c600154cd1906871b10"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810216    2551 flags.go:64] FLAG: --pod-manifest-path=""
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810218    2551 flags.go:64] FLAG: --pod-max-pids="-1"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810221    2551 flags.go:64] FLAG: --pods-per-core="0"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810224    2551 flags.go:64] FLAG: --port="10250"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810227    2551 flags.go:64] FLAG: --protect-kernel-defaults="false"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810231    2551 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-06bf24eb36dd655e5"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810234    2551 flags.go:64] FLAG: --qos-reserved=""
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810237    2551 flags.go:64] FLAG: --read-only-port="10255"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810240    2551 flags.go:64] FLAG: --register-node="true"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810243    2551 flags.go:64] FLAG: --register-with-taints=""
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810246    2551 flags.go:64] FLAG: --registry-burst="10"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810255    2551 flags.go:64] FLAG: --registry-qps="5"
Sep 11 13:19:28.817141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810258    2551 flags.go:64] FLAG: --reserved-cpus=""
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810261    2551 flags.go:64] FLAG: --reserved-memory=""
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810265    2551 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810268    2551 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810271    2551 flags.go:64] FLAG: --rotate-certificates="false"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810273    2551 flags.go:64] FLAG: --rotate-server-certificates="false"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810276    2551 flags.go:64] FLAG: --runonce="false"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810279    2551 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810282    2551 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810285    2551 flags.go:64] FLAG: --seccomp-default="false"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810288    2551 flags.go:64] FLAG: --serialize-image-pulls="true"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810290    2551 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810293    2551 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810296    2551 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810299    2551 flags.go:64] FLAG: --storage-driver-password="root"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810302    2551 flags.go:64] FLAG: --storage-driver-secure="false"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810305    2551 flags.go:64] FLAG: --storage-driver-table="stats"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810308    2551 flags.go:64] FLAG: --storage-driver-user="root"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810310    2551 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810313    2551 flags.go:64] FLAG: --sync-frequency="1m0s"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810316    2551 flags.go:64] FLAG: --system-cgroups=""
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810319    2551 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810324    2551 flags.go:64] FLAG: --system-reserved-cgroup=""
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810327    2551 flags.go:64] FLAG: --tls-cert-file=""
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810331    2551 flags.go:64] FLAG: --tls-cipher-suites="[]"
Sep 11 13:19:28.817750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810338    2551 flags.go:64] FLAG: --tls-min-version=""
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810341    2551 flags.go:64] FLAG: --tls-private-key-file=""
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810343    2551 flags.go:64] FLAG: --topology-manager-policy="none"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810346    2551 flags.go:64] FLAG: --topology-manager-policy-options=""
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810349    2551 flags.go:64] FLAG: --topology-manager-scope="container"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810352    2551 flags.go:64] FLAG: --v="2"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810355    2551 flags.go:64] FLAG: --version="false"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810365    2551 flags.go:64] FLAG: --vmodule=""
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810369    2551 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.810372    2551 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810494    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810498    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810502    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810504    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810507    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810511    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810513    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810519    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810523    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810526    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810528    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810531    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.818346 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810534    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810537    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810539    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810542    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810545    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810547    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810550    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810553    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810556    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810559    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810562    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810565    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810567    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810570    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810573    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810575    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810578    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810581    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810590    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810593    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.818904 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810595    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810598    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810601    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810603    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810606    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810608    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810611    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810615    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810618    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810621    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810623    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810641    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810644    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810647    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810650    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810653    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810656    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810658    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810661    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810664    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.819405 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810666    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810671    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810674    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810676    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810679    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810682    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810685    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810687    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810690    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810693    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810695    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810704    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810707    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810710    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810712    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810715    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810717    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810720    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810723    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.819946 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810727    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810731    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810734    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810737    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810740    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810743    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810746    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810749    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810752    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810755    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810757    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810760    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810763    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810765    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810769    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810772    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810775    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810778    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810780    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.820412 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810783    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810785    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810789    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810791    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810794    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810796    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810800    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810802    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.810805    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.811348    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.818320    2551 server.go:529] "Kubelet version" kubeletVersion="v1.34.9"
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.818336    2551 server.go:531] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.819172    2551 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.819179    2551 watchdog_linux.go:137] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Sep 11 13:19:28.820909 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819250    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819255    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819259    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819263    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819266    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819269    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819272    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819275    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819278    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819281    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819284    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819287    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819290    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819293    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819296    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819299    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819301    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819304    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819307    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819310    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.821271 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819312    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819315    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819317    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819320    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819322    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819325    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819328    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819331    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819333    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819337    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819340    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819343    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819345    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819348    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819351    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819354    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819356    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819359    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819361    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819364    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.821766 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819367    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819369    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819372    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819374    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819379    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819382    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819385    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819388    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819391    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819394    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819397    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819399    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819403    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819407    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819410    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819414    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819416    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819419    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819423    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.822255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819426    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819429    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819432    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819436    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819439    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819442    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819445    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819448    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819451    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819454    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819457    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819459    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819462    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819465    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819467    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819470    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819472    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819475    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819478    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819480    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.822746 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819483    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819485    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819488    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819490    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819493    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819496    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819498    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819501    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819503    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819505    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819508    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819511    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819513    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819516    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819519    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819521    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819525    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819527    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819530    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.823264 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819533    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.819538    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819678    2551 feature_gate.go:328] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819684    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819687    2551 feature_gate.go:328] unrecognized feature gate: HighlyAvailableArbiter
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819690    2551 feature_gate.go:328] unrecognized feature gate: NewOLMBoxCutterRuntime
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819693    2551 feature_gate.go:328] unrecognized feature gate: AzureClusterHostedDNSInstall
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819696    2551 feature_gate.go:328] unrecognized feature gate: SignatureStores
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819699    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNS
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819702    2551 feature_gate.go:328] unrecognized feature gate: MixedCPUsAllocation
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819704    2551 feature_gate.go:328] unrecognized feature gate: VSphereMixedNodeEnv
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819707    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819709    2551 feature_gate.go:328] unrecognized feature gate: ImageStreamImportMode
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819712    2551 feature_gate.go:328] unrecognized feature gate: InsightsOnDemandDataGather
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819715    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIController
Sep 11 13:19:28.823757 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819717    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesvSphere
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819720    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstallIBMCloud
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819722    2551 feature_gate.go:328] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819727    2551 feature_gate.go:351] Setting GA feature gate VolumeAttributesClass=true. It will be removed in a future release.
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819730    2551 feature_gate.go:328] unrecognized feature gate: BuildCSIVolumes
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819733    2551 feature_gate.go:328] unrecognized feature gate: EtcdBackendQuota
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819736    2551 feature_gate.go:328] unrecognized feature gate: MetricsCollectionProfiles
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819739    2551 feature_gate.go:328] unrecognized feature gate: GCPDualStackInstall
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819741    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNS
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819745    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPI
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819747    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAzure
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819750    2551 feature_gate.go:328] unrecognized feature gate: OVNObservability
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819752    2551 feature_gate.go:328] unrecognized feature gate: BootImageSkewEnforcement
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819755    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiDisk
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819758    2551 feature_gate.go:328] unrecognized feature gate: AzureWorkloadIdentity
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819762    2551 feature_gate.go:328] unrecognized feature gate: GatewayAPIWithoutOLM
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819765    2551 feature_gate.go:328] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819768    2551 feature_gate.go:349] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819772    2551 feature_gate.go:328] unrecognized feature gate: IrreconcilableMachineConfig
Sep 11 13:19:28.824127 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819775    2551 feature_gate.go:328] unrecognized feature gate: AWSDedicatedHosts
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819778    2551 feature_gate.go:328] unrecognized feature gate: OnPremDNSRecords
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819780    2551 feature_gate.go:328] unrecognized feature gate: PinnedImages
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819783    2551 feature_gate.go:328] unrecognized feature gate: MultiArchInstallAzure
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819785    2551 feature_gate.go:328] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819788    2551 feature_gate.go:328] unrecognized feature gate: NetworkDiagnosticsConfig
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819791    2551 feature_gate.go:328] unrecognized feature gate: DualReplica
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819793    2551 feature_gate.go:328] unrecognized feature gate: AWSDualStackInstall
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819796    2551 feature_gate.go:328] unrecognized feature gate: DNSNameResolver
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819799    2551 feature_gate.go:328] unrecognized feature gate: NoRegistryClusterInstall
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819801    2551 feature_gate.go:328] unrecognized feature gate: AzureMultiDisk
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819804    2551 feature_gate.go:328] unrecognized feature gate: CPMSMachineNamePrefix
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819806    2551 feature_gate.go:328] unrecognized feature gate: OpenShiftPodSecurityAdmission
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819810    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerificationPKI
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819812    2551 feature_gate.go:328] unrecognized feature gate: AdminNetworkPolicy
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819815    2551 feature_gate.go:328] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819818    2551 feature_gate.go:328] unrecognized feature gate: AutomatedEtcdBackup
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819820    2551 feature_gate.go:328] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819823    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryptionProvider
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819825    2551 feature_gate.go:328] unrecognized feature gate: Example2
Sep 11 13:19:28.824597 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819828    2551 feature_gate.go:328] unrecognized feature gate: ClusterMonitoringConfig
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819830    2551 feature_gate.go:328] unrecognized feature gate: VSphereHostVMGroupZonal
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819833    2551 feature_gate.go:328] unrecognized feature gate: ImageModeStatusReporting
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819836    2551 feature_gate.go:328] unrecognized feature gate: AzureDualStackInstall
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819838    2551 feature_gate.go:328] unrecognized feature gate: ExternalSnapshotMetadata
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819841    2551 feature_gate.go:328] unrecognized feature gate: GCPClusterHostedDNSInstall
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819843    2551 feature_gate.go:328] unrecognized feature gate: NutanixMultiSubnets
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819846    2551 feature_gate.go:328] unrecognized feature gate: OSStreams
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819851    2551 feature_gate.go:328] unrecognized feature gate: RouteAdvertisements
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819854    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesCPMS
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819857    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImagesAWS
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819859    2551 feature_gate.go:328] unrecognized feature gate: NewOLM
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819862    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDC
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819864    2551 feature_gate.go:328] unrecognized feature gate: ProvisioningRequestAvailable
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819867    2551 feature_gate.go:328] unrecognized feature gate: AlibabaPlatform
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819870    2551 feature_gate.go:328] unrecognized feature gate: Example
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819872    2551 feature_gate.go:328] unrecognized feature gate: NetworkSegmentation
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819875    2551 feature_gate.go:328] unrecognized feature gate: MachineConfigNodes
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819878    2551 feature_gate.go:328] unrecognized feature gate: PreconfiguredUDNAddresses
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819880    2551 feature_gate.go:328] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Sep 11 13:19:28.825101 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819883    2551 feature_gate.go:328] unrecognized feature gate: VolumeGroupSnapshot
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819885    2551 feature_gate.go:328] unrecognized feature gate: AzureDedicatedHosts
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819888    2551 feature_gate.go:328] unrecognized feature gate: ShortCertRotation
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819890    2551 feature_gate.go:328] unrecognized feature gate: GCPCustomAPIEndpoints
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819893    2551 feature_gate.go:328] unrecognized feature gate: ManagedBootImages
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819896    2551 feature_gate.go:328] unrecognized feature gate: UpgradeStatus
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819898    2551 feature_gate.go:328] unrecognized feature gate: EventTTL
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819901    2551 feature_gate.go:328] unrecognized feature gate: MultiDiskSetup
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819904    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIInstall
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819906    2551 feature_gate.go:328] unrecognized feature gate: NewOLMOwnSingleNamespace
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819909    2551 feature_gate.go:328] unrecognized feature gate: ClusterVersionOperatorConfiguration
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819912    2551 feature_gate.go:328] unrecognized feature gate: VSphereMultiNetworks
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819915    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIMigration
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819917    2551 feature_gate.go:328] unrecognized feature gate: AdditionalRoutingCapabilities
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819920    2551 feature_gate.go:328] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819923    2551 feature_gate.go:328] unrecognized feature gate: BootcNodeManagement
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819925    2551 feature_gate.go:328] unrecognized feature gate: ClusterAPIMachineManagement
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819928    2551 feature_gate.go:328] unrecognized feature gate: CRDCompatibilityRequirementOperator
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819930    2551 feature_gate.go:328] unrecognized feature gate: KMSEncryption
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819933    2551 feature_gate.go:328] unrecognized feature gate: NetworkLiveMigration
Sep 11 13:19:28.825585 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819937    2551 feature_gate.go:328] unrecognized feature gate: NewOLMPreflightPermissionChecks
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819939    2551 feature_gate.go:328] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819942    2551 feature_gate.go:328] unrecognized feature gate: InsightsConfig
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819945    2551 feature_gate.go:328] unrecognized feature gate: SigstoreImageVerification
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819949    2551 feature_gate.go:351] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819953    2551 feature_gate.go:328] unrecognized feature gate: AWSClusterHostedDNSInstall
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:28.819956    2551 feature_gate.go:328] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.819961    2551 feature_gate.go:384] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false EventedPLEG:false ImageVolume:true KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:false MutatingAdmissionPolicy:false ProcMountType:true RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true TranslateStreamCloseWebsocketRequests:false UserNamespacesPodSecurityStandards:true UserNamespacesSupport:true VolumeAttributesClass:true]}
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.820076    2551 server.go:962] "Client rotation is on, will bootstrap in background"
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.822406    2551 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.823483    2551 server.go:1019] "Starting client certificate rotation"
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.823585    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:19:28.826088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.823617    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Sep 11 13:19:28.826862 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.826846    2551 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:19:28.828067 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.828050    2551 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Sep 11 13:19:28.841961 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.841937    2551 log.go:25] "Validated CRI v1 runtime API"
Sep 11 13:19:28.845130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.845115    2551 log.go:25] "Validated CRI v1 image API"
Sep 11 13:19:28.845971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.845956    2551 server.go:1429] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Sep 11 13:19:28.851946 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.851921    2551 fs.go:135] Filesystem UUIDs: map[14d4a9ef-8b19-46c0-8784-7dc6e07c5c14:/dev/nvme0n1p4 58b9c956-13a9-4782-9893-ed357a6c5d5c:/dev/nvme0n1p3 7B77-95E7:/dev/nvme0n1p2]
Sep 11 13:19:28.852023 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.851946    2551 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:22 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:24 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:31 fsType:tmpfs blockSize:0} composefs_0-33:{mountpoint:/ major:0 minor:33 fsType:overlay blockSize:0}]
Sep 11 13:19:28.854313 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.854093    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:19:28.859685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.859557    2551 manager.go:216] Machine: {Timestamp:2026-09-11 13:19:28.858158582 +0000 UTC m=+0.359063024 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3102452 MemoryCapacity:33164476416 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2a17fbc1b36bf3320f4bcd0c1459fd SystemUUID:ec2a17fb-c1b3-6bf3-320f-4bcd0c1459fd BootID:9a72cb6c-8b90-401b-a494-e9af1a7640f3 Filesystems:[{Device:composefs_0-33 DeviceMajor:0 DeviceMinor:33 Capacity:6111232 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:31 Capacity:16582238208 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:22 Capacity:16582238208 Type:vfs Inodes:4048398 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:24 Capacity:6632898560 Type:vfs Inodes:819200 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:f3:5e:cd:fe:bb Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:f3:5e:cd:fe:bb Speed:0 Mtu:9001} {Name:ovs-system MacAddress:0a:b9:4a:13:3d:db Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:33164476416 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:37486592 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Sep 11 13:19:28.859685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.859682    2551 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Sep 11 13:19:28.859823 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.859766    2551 manager.go:232] Version: {KernelVersion:5.14.0-570.140.1.el9_6.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.6.20260908-1 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Sep 11 13:19:28.863170 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.863148    2551 container_manager_linux.go:273] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Sep 11 13:19:28.863304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.863173    2551 container_manager_linux.go:278] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-50.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Sep 11 13:19:28.864448 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.864437    2551 topology_manager.go:138] "Creating topology manager with none policy"
Sep 11 13:19:28.864482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.864451    2551 container_manager_linux.go:309] "Creating device plugin manager"
Sep 11 13:19:28.864482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.864459    2551 manager.go:141] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:19:28.865125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.865115    2551 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Sep 11 13:19:28.865954 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.865945    2551 container_manager_linux.go:318] "Creating Dynamic Resource Allocation (DRA) manager"
Sep 11 13:19:28.869643 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869621    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:19:28.869760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869751    2551 server.go:1240] "Using root directory" path="/var/lib/kubelet"
Sep 11 13:19:28.869805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869780    2551 kubelet.go:486] "Attempting to sync node with API server"
Sep 11 13:19:28.869805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869790    2551 kubelet.go:387] "Adding static pod path" path="/etc/kubernetes/manifests"
Sep 11 13:19:28.869805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869801    2551 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Sep 11 13:19:28.869913 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869811    2551 kubelet.go:398] "Adding apiserver pod source"
Sep 11 13:19:28.869913 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.869820    2551 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Sep 11 13:19:28.870186 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.870172    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:19:28.870231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.870198    2551 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Sep 11 13:19:28.872909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.872891    2551 kuberuntime_manager.go:301] "Container runtime initialized" containerRuntime="cri-o" version="1.34.12-5.rhaos4.21.gitd1ec63c.el9" apiVersion="v1"
Sep 11 13:19:28.874974 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.874960    2551 kubelet.go:960] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Sep 11 13:19:28.875035 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.874991    2551 kubelet.go:984] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Sep 11 13:19:28.876597 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876586    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Sep 11 13:19:28.876674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876606    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Sep 11 13:19:28.876674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876615    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Sep 11 13:19:28.876674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876621    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Sep 11 13:19:28.876674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876657    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Sep 11 13:19:28.876674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876664    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876678    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876684    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876691    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876696    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876705    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Sep 11 13:19:28.876807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.876710    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Sep 11 13:19:28.877551 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.877538    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Sep 11 13:19:28.877551 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.877548    2551 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Sep 11 13:19:28.881522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.881506    2551 server.go:1268] "Started kubelet"
Sep 11 13:19:28.881624 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.881589    2551 server.go:180] "Starting to listen" address="0.0.0.0" port=10250
Sep 11 13:19:28.881697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.881608    2551 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Sep 11 13:19:28.881767 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.881698    2551 server_v1.go:49] "podresources" method="list" useActivePods=true
Sep 11 13:19:28.882170 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.882151    2551 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-50.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Sep 11 13:19:28.882245 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.882168    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Node"
Sep 11 13:19:28.882245 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.882221    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.Service"
Sep 11 13:19:28.882574 ip-10-0-141-50 systemd[1]: Started Kubernetes Kubelet.
Sep 11 13:19:28.882980 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.882963    2551 server.go:249] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Sep 11 13:19:28.884310 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.884290    2551 server.go:310] "Adding debug handlers to kubelet server"
Sep 11 13:19:28.888649 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.888611    2551 kubelet.go:1638] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Sep 11 13:19:28.890344 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.890326    2551 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Sep 11 13:19:28.890472 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.889231    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-50.ec2.internal.18d446c629311ce6  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-50.ec2.internal,UID:ip-10-0-141-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-50.ec2.internal,},FirstTimestamp:2026-09-11 13:19:28.881482982 +0000 UTC m=+0.382387422,LastTimestamp:2026-09-11 13:19:28.881482982 +0000 UTC m=+0.382387422,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-50.ec2.internal,}"
Sep 11 13:19:28.890893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.890874    2551 fs_resource_analyzer.go:67] "Starting FS ResourceAnalyzer"
Sep 11 13:19:28.891072 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.891050    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:28.891577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891549    2551 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Sep 11 13:19:28.891577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891551    2551 volume_manager.go:311] "The desired_state_of_world populator starts"
Sep 11 13:19:28.891782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891580    2551 volume_manager.go:313] "Starting Kubelet Volume Manager"
Sep 11 13:19:28.891782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891682    2551 reconstruct.go:97] "Volume reconstruction finished"
Sep 11 13:19:28.891782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891690    2551 reconciler.go:29] "Reconciler: start to sync state"
Sep 11 13:19:28.892010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.891993    2551 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Sep 11 13:19:28.892010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892011    2551 factory.go:55] Registering systemd factory
Sep 11 13:19:28.892126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892021    2551 factory.go:223] Registration of the systemd container factory successfully
Sep 11 13:19:28.892269 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892252    2551 factory.go:153] Registering CRI-O factory
Sep 11 13:19:28.892306 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892272    2551 factory.go:223] Registration of the crio container factory successfully
Sep 11 13:19:28.892338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892311    2551 factory.go:103] Registering Raw factory
Sep 11 13:19:28.892338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892337    2551 manager.go:1196] Started watching for new ooms in manager
Sep 11 13:19:28.892901 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.892887    2551 manager.go:349] Starting recovery of all containers
Sep 11 13:19:28.898932 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.898905    2551 reflector.go:205] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.CSIDriver"
Sep 11 13:19:28.899036 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.898923    2551 controller.go:145] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-50.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Sep 11 13:19:28.904312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.904288    2551 manager.go:354] Recovery completed
Sep 11 13:19:28.910127 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.909966    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:28.912675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.912645    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:28.912757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.912688    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:28.912757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.912699    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:28.913187 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.913172    2551 cpu_manager.go:222] "Starting CPU manager" policy="none"
Sep 11 13:19:28.913268 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.913188    2551 cpu_manager.go:223] "Reconciling" reconcilePeriod="10s"
Sep 11 13:19:28.913268 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.913211    2551 state_mem.go:36] "Initialized new in-memory state store"
Sep 11 13:19:28.913268 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.913178    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:28.914158 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.914141    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:28.914219 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.914173    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:28.914219 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.914187    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:28.914693 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.914595    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-50.ec2.internal.18d446c62b0d10af  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-50.ec2.internal,UID:ip-10-0-141-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-50.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-50.ec2.internal,},FirstTimestamp:2026-09-11 13:19:28.912674991 +0000 UTC m=+0.413579435,LastTimestamp:2026-09-11 13:19:28.912674991 +0000 UTC m=+0.413579435,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-50.ec2.internal,}"
Sep 11 13:19:28.915045 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.915029    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m2s2h"
Sep 11 13:19:28.915295 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.915284    2551 policy_none.go:49] "None policy: Start"
Sep 11 13:19:28.915332 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.915300    2551 memory_manager.go:187] "Starting memorymanager" policy="None"
Sep 11 13:19:28.915332 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.915310    2551 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Sep 11 13:19:28.917243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.917231    2551 policy_none.go:47] "Start"
Sep 11 13:19:28.924098 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.924079    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-m2s2h"
Sep 11 13:19:28.924492 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.924432    2551 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-50.ec2.internal.18d446c62b0d5901  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-50.ec2.internal,UID:ip-10-0-141-50.ec2.internal,APIVersion:,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-50.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-50.ec2.internal,},FirstTimestamp:2026-09-11 13:19:28.912693505 +0000 UTC m=+0.413597946,LastTimestamp:2026-09-11 13:19:28.912693505 +0000 UTC m=+0.413597946,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-50.ec2.internal,}"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953212    2551 manager.go:337] "Starting Device Plugin manager"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.953250    2551 manager.go:513] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953263    2551 server.go:87] "Starting device plugin registration server"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953621    2551 eviction_manager.go:189] "Eviction manager: starting control loop"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953647    2551 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953700    2551 plugin_watcher.go:51] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953780    2551 plugin_manager.go:116] "The desired_state_of_world populator (plugin watcher) starts"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.953788    2551 plugin_manager.go:118] "Starting Kubelet Plugin Manager"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.954346    2551 eviction_manager.go:267] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.954384    2551 eviction_manager.go:292] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.956232    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.957470    2551 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.957521    2551 status_manager.go:244] "Starting to sync pod status with apiserver"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.957545    2551 kubelet.go:2448] "Starting kubelet main sync loop"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:28.957654    2551 kubelet.go:2472] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Sep 11 13:19:28.967470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:28.959945    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:19:29.054506 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.054394    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:29.055538 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.055520    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:29.055666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.055550    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:29.055666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.055575    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:29.055666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.055620    2551 kubelet_node_status.go:78] "Attempting to register node" node="ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.058664 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.058645    2551 kubelet.go:2535] "SyncLoop ADD" source="file" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"]
Sep 11 13:19:29.058725 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.058715    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:29.060109 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.060090    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:29.060189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.060121    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:29.060189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.060135    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:29.062303 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.062290    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:29.062462 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.062449    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.062496 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.062476    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:29.063067 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063050    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:29.063147 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063075    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:29.063147 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063096    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:29.063372 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063353    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:29.063467 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063382    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:29.063467 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.063394    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:29.065535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.065519    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.065601 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.065552    2551 kubelet_node_status.go:400] "Setting node annotation to enable volume controller attach/detach"
Sep 11 13:19:29.066261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.066247    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientMemory"
Sep 11 13:19:29.066333 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.066287    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasNoDiskPressure"
Sep 11 13:19:29.066333 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.066312    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeHasSufficientPID"
Sep 11 13:19:29.067013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.066991    2551 kubelet_node_status.go:81] "Successfully registered node" node="ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.067060 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.067020    2551 kubelet_node_status.go:536] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-50.ec2.internal\": node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.092351 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.092315    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb3cc3d67cb8f0daa2eb5dc040e5c1e9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"eb3cc3d67cb8f0daa2eb5dc040e5c1e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.092351 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.092346    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-50.ec2.internal\" not found" node="ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.092508 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.092368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.092508 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.092396    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.093608 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.093589    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.096867 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.096852    2551 kubelet.go:3244] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-50.ec2.internal\" not found" node="ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192589    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192643    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192856 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192668    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb3cc3d67cb8f0daa2eb5dc040e5c1e9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"eb3cc3d67cb8f0daa2eb5dc040e5c1e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192856 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/eb3cc3d67cb8f0daa2eb5dc040e5c1e9-config\") pod \"kube-apiserver-proxy-ip-10-0-141-50.ec2.internal\" (UID: \"eb3cc3d67cb8f0daa2eb5dc040e5c1e9\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192856 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192700    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.192856 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.192705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4c22a64c50251595bc078c42e30f8c3-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal\" (UID: \"f4c22a64c50251595bc078c42e30f8c3\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.193660 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.193636    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.294078 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.294045    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.394842 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.394773    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.395896 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.395880    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.399563 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.399548    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:29.495927 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.495889    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.596340 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.596299    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.696838 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.696758    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.797101 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.797074    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.823227 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.823204    2551 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Sep 11 13:19:29.823578 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.823332    2551 reflector.go:571] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:160" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:160: Unexpected watch close - watch lasted less than a second and no items received"
Sep 11 13:19:29.897672 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:29.897648    2551 kubelet_node_status.go:454] "Error getting the current node from lister" err="node \"ip-10-0-141-50.ec2.internal\" not found"
Sep 11 13:19:29.928943 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.928907    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-09-10 13:14:28 +0000 UTC" deadline="2028-04-09 00:20:32.321812793 +0000 UTC"
Sep 11 13:19:29.928943 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.928940    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13811h1m2.392876259s"
Sep 11 13:19:29.933297 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.933283    2551 reflector.go:439] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:19:29.970716 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.970655    2551 reflector.go:439] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:19:29.991761 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:29.991734    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:30.007136 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.007115    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:19:30.008764 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.008752    2551 kubelet.go:3248] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal"
Sep 11 13:19:30.022704 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.022684    2551 warnings.go:110] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Sep 11 13:19:30.059276 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:30.059241    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeb3cc3d67cb8f0daa2eb5dc040e5c1e9.slice/crio-9641cc33fc49fb62e0b885bdfa5d2e3e4aeb945e1dd7002dfbb10d6cd0f37f4d WatchSource:0}: Error finding container 9641cc33fc49fb62e0b885bdfa5d2e3e4aeb945e1dd7002dfbb10d6cd0f37f4d: Status 404 returned error can't find the container with id 9641cc33fc49fb62e0b885bdfa5d2e3e4aeb945e1dd7002dfbb10d6cd0f37f4d
Sep 11 13:19:30.059471 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:30.059456    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4c22a64c50251595bc078c42e30f8c3.slice/crio-e437487909b6be1fce117efbe836415efca0c58a4c957fe90d7a2cbf7a152f85 WatchSource:0}: Error finding container e437487909b6be1fce117efbe836415efca0c58a4c957fe90d7a2cbf7a152f85: Status 404 returned error can't find the container with id e437487909b6be1fce117efbe836415efca0c58a4c957fe90d7a2cbf7a152f85
Sep 11 13:19:30.067472 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.067453    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:19:30.315869 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.315793    2551 reflector.go:439] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:19:30.870651 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.870610    2551 apiserver.go:52] "Watching apiserver"
Sep 11 13:19:30.877355 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.877326    2551 reflector.go:439] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Sep 11 13:19:30.880344 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.880309    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r","openshift-cluster-node-tuning-operator/tuned-pr4dg","openshift-dns/node-resolver-gh9gl","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal","openshift-multus/network-metrics-daemon-r9tc6","openshift-network-diagnostics/network-check-target-fm9hc","kube-system/konnectivity-agent-6z4jd","openshift-image-registry/node-ca-9wzjp","openshift-multus/multus-additional-cni-plugins-snvtd","openshift-multus/multus-cf7rg","openshift-network-operator/iptables-alerter-w249f","openshift-ovn-kubernetes/ovnkube-node-rqzls","kube-system/global-pull-secret-syncer-zgb2x"]
Sep 11 13:19:30.882936 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.882910    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.885245 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.885223    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.887327 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.887309    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:30.888182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.888162    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Sep 11 13:19:30.888385 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.888371    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.888513 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.888500    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-8mc9b\""
Sep 11 13:19:30.888724 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.888706    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.889718 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.889571    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:30.889718 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:30.889670    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:30.891010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.890961    2551 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Sep 11 13:19:30.892119 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.891579    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.892119 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.892047    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:30.892119 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.892074    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.892119 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:30.892106    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:30.892383 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.892155    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-kfjg5\""
Sep 11 13:19:30.894182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.894163    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:30.896619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.896597    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:30.898319 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.898299    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.898898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.898523    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.898898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.898624    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-4b6fr\""
Sep 11 13:19:30.899041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.898898    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:30.899285 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-registration-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.899285 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.899285 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899256    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-var-lib-kubelet\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899481 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899300    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j88sm\" (UniqueName: \"kubernetes.io/projected/c9b50cf3-1019-4849-849b-ac93610a294e-kube-api-access-j88sm\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.899481 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899414    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.899579 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899511    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899648 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899576    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-tuned\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899704 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899664    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5p77b\" (UniqueName: \"kubernetes.io/projected/bc68299e-de59-4d00-a185-b9efb86dc38e-kube-api-access-5p77b\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899697    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-device-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.899754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899724    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysconfig\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899754    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-run\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.899844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817a83e4-76b4-4ac1-8afe-27c3beed2530-tmp-dir\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:30.899844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:30.900173 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.899834    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:30.900237 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900216    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b863f06e-24b9-4893-80f4-d6923e658891-agent-certs\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:30.900299 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900282    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b863f06e-24b9-4893-80f4-d6923e658891-konnectivity-ca\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:30.900347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900321    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-sys-fs\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.900396 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900363    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-modprobe-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900392    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-tmp\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900421    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-lib-modules\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900538 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900452    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-host\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900538 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900505    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-kubernetes\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900652 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900575    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/817a83e4-76b4-4ac1-8afe-27c3beed2530-hosts-file\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:30.900747 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900715    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z85ph\" (UniqueName: \"kubernetes.io/projected/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-kube-api-access-z85ph\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:30.900816 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900801    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-conf\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900849    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-systemd\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.900935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900906    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-sys\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:30.901031 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.900948    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6ml6t\" (UniqueName: \"kubernetes.io/projected/817a83e4-76b4-4ac1-8afe-27c3beed2530-kube-api-access-6ml6t\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:30.901100 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.901029    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-socket-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:30.902935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.902911    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Sep 11 13:19:30.904148 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.904127    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:30.904246 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.904156    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.905442 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.905420    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Sep 11 13:19:30.906389 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.905985    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:30.908757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.908737    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:30.910934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.910914    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:30.911028 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:30.910981    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:30.926136 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926034    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"default-dockercfg-km84p\""
Sep 11 13:19:30.926299 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926281    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Sep 11 13:19:30.926736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926540    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-2q9k5\""
Sep 11 13:19:30.926736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926557    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Sep 11 13:19:30.926896 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926769    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.926896 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926837    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Sep 11 13:19:30.926896 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.926839    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.930440 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.930403    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-tnvj2\""
Sep 11 13:19:30.930574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.930555    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Sep 11 13:19:30.930962 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.930942    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.931146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.931130    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932281    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-7h5kz\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932339    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932285    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"default-dockercfg-sk2t2\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932285    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932490    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-8v24g\""
Sep 11 13:19:30.932629 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932530    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Sep 11 13:19:30.933068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932785    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Sep 11 13:19:30.933068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932816    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Sep 11 13:19:30.933068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.932844    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Sep 11 13:19:30.933068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.933039    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Sep 11 13:19:30.934184 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.934165    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.936733 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.936713    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Sep 11 13:19:30.957968 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.957946    2551 reflector.go:439] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Sep 11 13:19:30.962683 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.962615    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" event={"ID":"eb3cc3d67cb8f0daa2eb5dc040e5c1e9","Type":"ContainerStarted","Data":"9641cc33fc49fb62e0b885bdfa5d2e3e4aeb945e1dd7002dfbb10d6cd0f37f4d"}
Sep 11 13:19:30.965699 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.965673    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"f4c22a64c50251595bc078c42e30f8c3","Type":"ContainerStarted","Data":"e437487909b6be1fce117efbe836415efca0c58a4c957fe90d7a2cbf7a152f85"}
Sep 11 13:19:30.992823 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:30.992795    2551 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Sep 11 13:19:31.002112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002069    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5p77b\" (UniqueName: \"kubernetes.io/projected/bc68299e-de59-4d00-a185-b9efb86dc38e-kube-api-access-5p77b\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.002112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002117    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-device-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.002112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002148    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-kubelet\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002186    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-log-socket\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002218    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-serviceca\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002242    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cni-binary-copy\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002263    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-netns\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002286    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002312    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-sys-fs\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002335    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09ce8773-3f3d-4122-ab10-c33cb17bb152-iptables-alerter-script\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002358    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-config\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002417    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-conf-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002461    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-multus-certs\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002487    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-hostroot\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002507    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-lib-modules\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-kubernetes\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002565    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-systemd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002598    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-script-lib\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002606    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-device-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002663    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-k8s-cni-cncf-io\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002701    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z85ph\" (UniqueName: \"kubernetes.io/projected/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-kube-api-access-z85ph\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002744    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2srl4\" (UniqueName: \"kubernetes.io/projected/09ce8773-3f3d-4122-ab10-c33cb17bb152-kube-api-access-2srl4\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-sys-fs\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002769    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002830    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8mwm\" (UniqueName: \"kubernetes.io/projected/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-kube-api-access-s8mwm\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.002865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002861    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-socket-dir-parent\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002886    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-systemd\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002912    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-sys\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002901    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-lib-modules\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002939    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6ml6t\" (UniqueName: \"kubernetes.io/projected/817a83e4-76b4-4ac1-8afe-27c3beed2530-kube-api-access-6ml6t\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002964    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-registration-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.002993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-ovn\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003017    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003027    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-kubernetes\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003042    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003090    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-slash\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003102    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-sys\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003122    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-node-log\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003147    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-host\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.003154    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003160    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-systemd\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003173    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.003550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003201    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-bin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003229    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-multus\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003255    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-etc-kubernetes\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003285    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003309    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-tuned\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003331    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09ce8773-3f3d-4122-ab10-c33cb17bb152-host-slash\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-netd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysconfig\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.003464    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:31.503432728 +0000 UTC m=+3.004337161 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003516    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-run\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003528    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysconfig\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003556    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817a83e4-76b4-4ac1-8afe-27c3beed2530-tmp-dir\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003596    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-dbus\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003620    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003617    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-system-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003676    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.004403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003722    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003729    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-registration-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003744    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b863f06e-24b9-4893-80f4-d6923e658891-agent-certs\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b863f06e-24b9-4893-80f4-d6923e658891-konnectivity-ca\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003833    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-netns\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003855    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-env-overrides\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003907    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-modprobe-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003926    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-tmp\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-system-cni-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003974    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sj6rf\" (UniqueName: \"kubernetes.io/projected/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-kube-api-access-sj6rf\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.003993    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cnibin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004012    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/817a83e4-76b4-4ac1-8afe-27c3beed2530-tmp-dir\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004025    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q28kj\" (UniqueName: \"kubernetes.io/projected/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-kube-api-access-q28kj\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004154    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-host\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004196    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/817a83e4-76b4-4ac1-8afe-27c3beed2530-hosts-file\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004240    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-var-lib-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004254    2551 swap_util.go:74] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Sep 11 13:19:31.005162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004310    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-etc-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004317    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-modprobe-d\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004353    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cnibin\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004378    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-daemon-config\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004411    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004471    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/817a83e4-76b4-4ac1-8afe-27c3beed2530-hosts-file\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004911    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-kubelet-config\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.004955    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-run\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005023    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-conf\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005104    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-socket-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005113    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/b863f06e-24b9-4893-80f4-d6923e658891-konnectivity-ca\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005158    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005184    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j88sm\" (UniqueName: \"kubernetes.io/projected/c9b50cf3-1019-4849-849b-ac93610a294e-kube-api-access-j88sm\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-bin\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005242    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-host\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005261    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vx8\" (UniqueName: \"kubernetes.io/projected/1657531f-635c-48c7-917d-30db46e0acfd-kube-api-access-c7vx8\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005266    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-sysctl-conf\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005284    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005292    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-etc-selinux\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005331    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1657531f-635c-48c7-917d-30db46e0acfd-ovn-node-metrics-cert\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005358    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-socket-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005442    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-var-lib-kubelet\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-systemd-units\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005510    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-kubelet\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005532    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/bc68299e-de59-4d00-a185-b9efb86dc38e-var-lib-kubelet\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.006842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.005595    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/c9b50cf3-1019-4849-849b-ac93610a294e-kubelet-dir\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.008509 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.008484    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-etc-tuned\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.008604 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.008535    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/bc68299e-de59-4d00-a185-b9efb86dc38e-tmp\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.016167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.016103    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/b863f06e-24b9-4893-80f4-d6923e658891-agent-certs\") pod \"konnectivity-agent-6z4jd\" (UID: \"b863f06e-24b9-4893-80f4-d6923e658891\") " pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:31.021147 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.021126    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:31.021147 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.021150    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:31.021294 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.021163    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:31.021647 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.021618    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:19:31.521597295 +0000 UTC m=+3.022501743 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:31.023792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.023772    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5p77b\" (UniqueName: \"kubernetes.io/projected/bc68299e-de59-4d00-a185-b9efb86dc38e-kube-api-access-5p77b\") pod \"tuned-pr4dg\" (UID: \"bc68299e-de59-4d00-a185-b9efb86dc38e\") " pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.023915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.023893    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z85ph\" (UniqueName: \"kubernetes.io/projected/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-kube-api-access-z85ph\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:31.025926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.025900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6ml6t\" (UniqueName: \"kubernetes.io/projected/817a83e4-76b4-4ac1-8afe-27c3beed2530-kube-api-access-6ml6t\") pod \"node-resolver-gh9gl\" (UID: \"817a83e4-76b4-4ac1-8afe-27c3beed2530\") " pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.038641 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.038603    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-j88sm\" (UniqueName: \"kubernetes.io/projected/c9b50cf3-1019-4849-849b-ac93610a294e-kube-api-access-j88sm\") pod \"aws-ebs-csi-driver-node-7zk2r\" (UID: \"c9b50cf3-1019-4849-849b-ac93610a294e\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.090309 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.090274    2551 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qxhd"
Sep 11 13:19:31.095445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.093593    2551 reflector.go:439] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:160"
Sep 11 13:19:31.102281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.102258    2551 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-4qxhd"
Sep 11 13:19:31.106277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106251    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.106391 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106303    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-netns\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.106391 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106330    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-env-overrides\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.106391 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106355    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-system-cni-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.106391 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106380    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sj6rf\" (UniqueName: \"kubernetes.io/projected/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-kube-api-access-sj6rf\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106397    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-netns\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106405    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cnibin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106457    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cnibin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106477    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q28kj\" (UniqueName: \"kubernetes.io/projected/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-kube-api-access-q28kj\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106502    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-var-lib-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106535    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-etc-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106559    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cnibin\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.106591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106583    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-daemon-config\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106617    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106655    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-kubelet-config\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106705    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-var-lib-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106716    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-kubelet-config\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106511    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-system-cni-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106754    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-etc-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106785    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cnibin\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106817    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-bin\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106841    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-openvswitch\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106842    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vx8\" (UniqueName: \"kubernetes.io/projected/1657531f-635c-48c7-917d-30db46e0acfd-kube-api-access-c7vx8\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106898    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-env-overrides\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1657531f-635c-48c7-917d-30db46e0acfd-ovn-node-metrics-cert\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106968    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-bin\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.106986    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-systemd-units\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107031    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-kubelet\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107070    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-kubelet\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107096    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-systemd-units\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107095    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-log-socket\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107138    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-serviceca\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107142    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-log-socket\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107163    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cni-binary-copy\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-netns\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107190    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-kubelet\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107222    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09ce8773-3f3d-4122-ab10-c33cb17bb152-iptables-alerter-script\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107244    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-kubelet\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107248    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-config\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107281    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107308    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-conf-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107324    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-daemon-config\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107331    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-multus-certs\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107356    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-hostroot\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107382    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-systemd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.107918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107392    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-netns\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-script-lib\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107432    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-k8s-cni-cncf-io\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107461    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2srl4\" (UniqueName: \"kubernetes.io/projected/09ce8773-3f3d-4122-ab10-c33cb17bb152-kube-api-access-2srl4\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107484    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107509    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8mwm\" (UniqueName: \"kubernetes.io/projected/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-kube-api-access-s8mwm\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107510    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-binary-copy\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107530    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107584    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-multus-certs\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107591    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-systemd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107647    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-hostroot\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107536    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-socket-dir-parent\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107754    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-config\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107761    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-tuning-conf-dir\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-ovn\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107806    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-serviceca\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107821    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-run-ovn\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.108858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107829    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-run-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.107966    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-socket-dir-parent\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108016    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-multus-conf-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108045    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108073    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108100    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-slash\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108108    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-cni-binary-copy\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-node-log\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108125    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108162    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-node-log\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108200    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-slash\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.108211    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108236    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-host\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.108262    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:31.608248443 +0000 UTC m=+3.109152876 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108202    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-run-k8s-cni-cncf-io\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-host\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108280    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.109645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108317    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-bin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108339    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-multus\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108361    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-etc-kubernetes\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108388    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09ce8773-3f3d-4122-ab10-c33cb17bb152-host-slash\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108401    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-multus\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-netd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108289    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/1657531f-635c-48c7-917d-30db46e0acfd-ovnkube-script-lib\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-dbus\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108465    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-etc-kubernetes\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-system-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108520    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/1657531f-635c-48c7-917d-30db46e0acfd-host-cni-netd\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108520    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/06617998-5730-4981-9d6d-1ccbc62c7ab5-dbus\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108468    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/09ce8773-3f3d-4122-ab10-c33cb17bb152-host-slash\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108464    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-host-var-lib-cni-bin\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108559    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/09ce8773-3f3d-4122-ab10-c33cb17bb152-iptables-alerter-script\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108550    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-system-cni-dir\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.110451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.108780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.111208 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.110755    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/1657531f-635c-48c7-917d-30db46e0acfd-ovn-node-metrics-cert\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.150154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.150083    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-2srl4\" (UniqueName: \"kubernetes.io/projected/09ce8773-3f3d-4122-ab10-c33cb17bb152-kube-api-access-2srl4\") pod \"iptables-alerter-w249f\" (UID: \"09ce8773-3f3d-4122-ab10-c33cb17bb152\") " pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.150297 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.150243    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vx8\" (UniqueName: \"kubernetes.io/projected/1657531f-635c-48c7-917d-30db46e0acfd-kube-api-access-c7vx8\") pod \"ovnkube-node-rqzls\" (UID: \"1657531f-635c-48c7-917d-30db46e0acfd\") " pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.150735 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.150713    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8mwm\" (UniqueName: \"kubernetes.io/projected/ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c-kube-api-access-s8mwm\") pod \"node-ca-9wzjp\" (UID: \"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c\") " pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.151142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.151111    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sj6rf\" (UniqueName: \"kubernetes.io/projected/d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d-kube-api-access-sj6rf\") pod \"multus-additional-cni-plugins-snvtd\" (UID: \"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d\") " pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.163269 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.163250    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q28kj\" (UniqueName: \"kubernetes.io/projected/b6b66705-ca0f-41f6-80fa-b6c4e478ccdb-kube-api-access-q28kj\") pod \"multus-cf7rg\" (UID: \"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb\") " pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.195838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.195808    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r"
Sep 11 13:19:31.204910 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.204886    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-pr4dg"
Sep 11 13:19:31.214854 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.214821    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-gh9gl"
Sep 11 13:19:31.220709 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.220685    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:31.228737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.228709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-9wzjp"
Sep 11 13:19:31.236976 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.236945    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-snvtd"
Sep 11 13:19:31.246744 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.246718    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-cf7rg"
Sep 11 13:19:31.255505 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.255486    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-w249f"
Sep 11 13:19:31.261273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.261246    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:31.511903 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.511827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:31.512056 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.511981    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:31.512056 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.512050    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:32.51203501 +0000 UTC m=+4.012939456 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:31.612787 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.612756    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:31.612950 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:31.612825    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:31.612950 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.612919    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:31.612950 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.612931    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:31.613087 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.612954    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:31.613087 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.612965    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:31.613087 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.612989    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:32.6129664 +0000 UTC m=+4.113870826 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:31.613087 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:31.613024    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:19:32.613012736 +0000 UTC m=+4.113917162 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:32.107343 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.107251    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:14:31 +0000 UTC" deadline="2028-04-05 13:27:55.940768099 +0000 UTC"
Sep 11 13:19:32.107343 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.107297    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13728h8m23.833475535s"
Sep 11 13:19:32.183441 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.183221    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbc68299e_de59_4d00_a185_b9efb86dc38e.slice/crio-a95ebd0f12733d8ea3bd8288f6384cf073d9fe85cdf3022295e5c489ea4d51db WatchSource:0}: Error finding container a95ebd0f12733d8ea3bd8288f6384cf073d9fe85cdf3022295e5c489ea4d51db: Status 404 returned error can't find the container with id a95ebd0f12733d8ea3bd8288f6384cf073d9fe85cdf3022295e5c489ea4d51db
Sep 11 13:19:32.185249 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.185219    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb863f06e_24b9_4893_80f4_d6923e658891.slice/crio-9f4832680403555468f75de636a2af0b202aafd25d621235201ca04e2de8cb5f WatchSource:0}: Error finding container 9f4832680403555468f75de636a2af0b202aafd25d621235201ca04e2de8cb5f: Status 404 returned error can't find the container with id 9f4832680403555468f75de636a2af0b202aafd25d621235201ca04e2de8cb5f
Sep 11 13:19:32.188996 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.188971    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1657531f_635c_48c7_917d_30db46e0acfd.slice/crio-0bd2cdeede069ace98a5b915e9ccb123f4bdf0d1051cc8929b92ea1e38f8326b WatchSource:0}: Error finding container 0bd2cdeede069ace98a5b915e9ccb123f4bdf0d1051cc8929b92ea1e38f8326b: Status 404 returned error can't find the container with id 0bd2cdeede069ace98a5b915e9ccb123f4bdf0d1051cc8929b92ea1e38f8326b
Sep 11 13:19:32.190189 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.190165    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee0bb0ee_3a05_4c9c_8191_0a9c1879af2c.slice/crio-9ad96a2013d49aef625d7a9f54a50eb0b400a5f49fd3a5aef59ebb0a005f2f81 WatchSource:0}: Error finding container 9ad96a2013d49aef625d7a9f54a50eb0b400a5f49fd3a5aef59ebb0a005f2f81: Status 404 returned error can't find the container with id 9ad96a2013d49aef625d7a9f54a50eb0b400a5f49fd3a5aef59ebb0a005f2f81
Sep 11 13:19:32.192807 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.192743    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09ce8773_3f3d_4122_ab10_c33cb17bb152.slice/crio-a04566b1a3c596cdb7c98655cd8d2f71da7805a32b9023b084117bc83d76a21c WatchSource:0}: Error finding container a04566b1a3c596cdb7c98655cd8d2f71da7805a32b9023b084117bc83d76a21c: Status 404 returned error can't find the container with id a04566b1a3c596cdb7c98655cd8d2f71da7805a32b9023b084117bc83d76a21c
Sep 11 13:19:32.194606 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.194509    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9b50cf3_1019_4849_849b_ac93610a294e.slice/crio-25a2c31b111280a766a0396cb61391d67e5700b5a700ad530a8f312bf3dd4e9f WatchSource:0}: Error finding container 25a2c31b111280a766a0396cb61391d67e5700b5a700ad530a8f312bf3dd4e9f: Status 404 returned error can't find the container with id 25a2c31b111280a766a0396cb61391d67e5700b5a700ad530a8f312bf3dd4e9f
Sep 11 13:19:32.194990 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.194970    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb6b66705_ca0f_41f6_80fa_b6c4e478ccdb.slice/crio-b77e94498520e59a906234506154514481e8b146363a91cb97f6d5b2ba36d399 WatchSource:0}: Error finding container b77e94498520e59a906234506154514481e8b146363a91cb97f6d5b2ba36d399: Status 404 returned error can't find the container with id b77e94498520e59a906234506154514481e8b146363a91cb97f6d5b2ba36d399
Sep 11 13:19:32.196201 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.196180    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod817a83e4_76b4_4ac1_8afe_27c3beed2530.slice/crio-bc754c3f34fb9b78cbc8a4d9a07cef44bf6965db4a8d4d7e94db141a772baa6f WatchSource:0}: Error finding container bc754c3f34fb9b78cbc8a4d9a07cef44bf6965db4a8d4d7e94db141a772baa6f: Status 404 returned error can't find the container with id bc754c3f34fb9b78cbc8a4d9a07cef44bf6965db4a8d4d7e94db141a772baa6f
Sep 11 13:19:32.197543 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:19:32.197521    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd8b69eb1_44c3_4ff1_8b28_b6e4d412c58d.slice/crio-8432c6dac4904b410ba1c092b47efff694305ec724fbb29f39b643dcf32ef0c6 WatchSource:0}: Error finding container 8432c6dac4904b410ba1c092b47efff694305ec724fbb29f39b643dcf32ef0c6: Status 404 returned error can't find the container with id 8432c6dac4904b410ba1c092b47efff694305ec724fbb29f39b643dcf32ef0c6
Sep 11 13:19:32.520500 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.520467    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:32.520674 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.520598    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:32.520750 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.520684    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:34.520666252 +0000 UTC m=+6.021570697 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:32.621712 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.621674    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:32.621882 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.621773    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:32.621940 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.621892    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:32.621990 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.621956    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:34.621936067 +0000 UTC m=+6.122840511 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:32.621990 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.621960    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:32.621990 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.621987    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:32.622146 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.622000    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:32.622146 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.622049    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:19:34.622036061 +0000 UTC m=+6.122940509 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:32.960590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.960398    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:32.960590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.960539    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:32.961232 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.961213    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:32.961345 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.961327    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:32.961437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.961390    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:32.961480 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:32.961458    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:32.988848 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:32.988807    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" event={"ID":"eb3cc3d67cb8f0daa2eb5dc040e5c1e9","Type":"ContainerStarted","Data":"f5b73c9ff60c96af6e80fe9b166d3ba5429bf79212b55b721d3893d7a1305909"}
Sep 11 13:19:33.006991 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.006932    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-50.ec2.internal" podStartSLOduration=4.006912062 podStartE2EDuration="4.006912062s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:33.006729953 +0000 UTC m=+4.507634426" watchObservedRunningTime="2026-09-11 13:19:33.006912062 +0000 UTC m=+4.507816530"
Sep 11 13:19:33.010393 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.010363    2551 generic.go:356] "Generic (PLEG): container finished" podID="f4c22a64c50251595bc078c42e30f8c3" containerID="ab8042effd08b785d11c0aa123188fcad90978d8cd18667d9060864cb1c08aec" exitCode=0
Sep 11 13:19:33.010518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.010436    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"f4c22a64c50251595bc078c42e30f8c3","Type":"ContainerDied","Data":"ab8042effd08b785d11c0aa123188fcad90978d8cd18667d9060864cb1c08aec"}
Sep 11 13:19:33.017661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.017609    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerStarted","Data":"8432c6dac4904b410ba1c092b47efff694305ec724fbb29f39b643dcf32ef0c6"}
Sep 11 13:19:33.036845 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.036748    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gh9gl" event={"ID":"817a83e4-76b4-4ac1-8afe-27c3beed2530","Type":"ContainerStarted","Data":"bc754c3f34fb9b78cbc8a4d9a07cef44bf6965db4a8d4d7e94db141a772baa6f"}
Sep 11 13:19:33.053177 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.052918    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w249f" event={"ID":"09ce8773-3f3d-4122-ab10-c33cb17bb152","Type":"ContainerStarted","Data":"a04566b1a3c596cdb7c98655cd8d2f71da7805a32b9023b084117bc83d76a21c"}
Sep 11 13:19:33.062618 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.062510    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"0bd2cdeede069ace98a5b915e9ccb123f4bdf0d1051cc8929b92ea1e38f8326b"}
Sep 11 13:19:33.069356 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.069325    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z4jd" event={"ID":"b863f06e-24b9-4893-80f4-d6923e658891","Type":"ContainerStarted","Data":"9f4832680403555468f75de636a2af0b202aafd25d621235201ca04e2de8cb5f"}
Sep 11 13:19:33.073661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.073613    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cf7rg" event={"ID":"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb","Type":"ContainerStarted","Data":"b77e94498520e59a906234506154514481e8b146363a91cb97f6d5b2ba36d399"}
Sep 11 13:19:33.080804 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.080779    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r" event={"ID":"c9b50cf3-1019-4849-849b-ac93610a294e","Type":"ContainerStarted","Data":"25a2c31b111280a766a0396cb61391d67e5700b5a700ad530a8f312bf3dd4e9f"}
Sep 11 13:19:33.087180 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.087154    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wzjp" event={"ID":"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c","Type":"ContainerStarted","Data":"9ad96a2013d49aef625d7a9f54a50eb0b400a5f49fd3a5aef59ebb0a005f2f81"}
Sep 11 13:19:33.105719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.105646    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr4dg" event={"ID":"bc68299e-de59-4d00-a185-b9efb86dc38e","Type":"ContainerStarted","Data":"a95ebd0f12733d8ea3bd8288f6384cf073d9fe85cdf3022295e5c489ea4d51db"}
Sep 11 13:19:33.108182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.108130    2551 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-09-10 13:14:31 +0000 UTC" deadline="2028-06-25 19:09:12.112845632 +0000 UTC"
Sep 11 13:19:33.108182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:33.108163    2551 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15677h49m39.004687724s"
Sep 11 13:19:34.132454 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.132416    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" event={"ID":"f4c22a64c50251595bc078c42e30f8c3","Type":"ContainerStarted","Data":"4a48c8443a223cbb8e8368b7913a1c79feb56140ddfec8939b84bcff8a522b88"}
Sep 11 13:19:34.544395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.544354    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:34.544562 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.544523    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:34.544658 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.544593    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:38.544572861 +0000 UTC m=+10.045477311 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:34.645435 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.645395    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:34.645606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.645461    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:34.645606 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.645598    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:34.645740 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.645619    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:34.645740 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.645647    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:34.645740 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.645708    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:19:38.645690076 +0000 UTC m=+10.146594511 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:34.646125 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.646099    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:34.646198 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.646163    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:38.646147586 +0000 UTC m=+10.147052016 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:34.959866 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.959746    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:34.960029 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.959882    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:34.960418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.960144    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:34.960418 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.960288    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:34.960418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:34.960153    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:34.960418 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:34.960386    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:36.958318 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:36.958281    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:36.958769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:36.958385    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:36.958769 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:36.958395    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:36.958769 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:36.958500    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:36.958769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:36.958285    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:36.958769 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:36.958591    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:38.579523 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.579486    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:38.580030 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.579735    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:38.580030 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.579805    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:46.579785336 +0000 UTC m=+18.080689807 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.680033    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.680137    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680252    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680279    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680292    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680329    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680381    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:19:46.68033398 +0000 UTC m=+18.181238410 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:38.680393 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.680404    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:19:46.680393447 +0000 UTC m=+18.181297874 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:38.959000 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.958781    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:38.959000 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.958896    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:38.959000 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.958975    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:38.959229 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.959088    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:38.959229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:38.959121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:38.959229 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:38.959184    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:40.958335 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:40.958245    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:40.958767 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:40.958392    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:40.958767 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:40.958385    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:40.958767 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:40.958495    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:40.958767 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:40.958536    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:40.958767 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:40.958622    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:42.959096 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:42.958845    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:42.959488 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:42.958845    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:42.959488 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:42.959182    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:42.959488 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:42.958888    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:42.959488 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:42.959301    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:42.959488 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:42.959394    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:44.958175 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:44.958146    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:44.958592 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:44.958146    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:44.958592 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:44.958259    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:44.958592 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:44.958146    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:44.958592 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:44.958357    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:44.958592 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:44.958421    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:46.639708 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.639669    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:46.640138 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.639863    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:46.640138 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.639943    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:02.639927229 +0000 UTC m=+34.140831656 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:19:46.740850 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.740815    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:46.740850 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.740856    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.740971    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.740985    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.740999    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.741013    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.741044    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:02.741026204 +0000 UTC m=+34.241930651 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:19:46.741073 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.741059    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:20:02.741052782 +0000 UTC m=+34.241957212 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:19:46.959016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.958886    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:46.959016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.958957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:46.959016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:46.958965    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:46.959283 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.959074    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:46.959283 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.959174    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:46.959283 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:46.959268    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:48.959226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:48.959198    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:48.959688 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:48.959303    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:48.959688 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:48.959393    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:48.959688 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:48.959488    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:48.959688 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:48.959541    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:48.959688 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:48.959657    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:50.165515 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.165231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r" event={"ID":"c9b50cf3-1019-4849-849b-ac93610a294e","Type":"ContainerStarted","Data":"af2d342ae65f3407693fb475cc695428442852cf480d598a767892d1d837b404"}
Sep 11 13:19:50.166821 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.166793    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-9wzjp" event={"ID":"ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c","Type":"ContainerStarted","Data":"5bee2dfd5bd32d64ab7f7b74da70e9e1eca3eb0921d9d35eec2e519123e8dcaf"}
Sep 11 13:19:50.168368 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.168344    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-pr4dg" event={"ID":"bc68299e-de59-4d00-a185-b9efb86dc38e","Type":"ContainerStarted","Data":"ef48b51fc0230a2e9a1e0fc59edeab581d98a10b4d6e98ddfa5416cd93bfbf9a"}
Sep 11 13:19:50.169960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.169933    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="604003f1de4c0b58c046495d48c05f344a6bed1bd436fbe8feff0f6df38175c6" exitCode=0
Sep 11 13:19:50.170051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.170005    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"604003f1de4c0b58c046495d48c05f344a6bed1bd436fbe8feff0f6df38175c6"}
Sep 11 13:19:50.171737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.171712    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-gh9gl" event={"ID":"817a83e4-76b4-4ac1-8afe-27c3beed2530","Type":"ContainerStarted","Data":"60c8cd4c896fa6bed214fa349b5319c1edeb7b0c9216e98bfd3ccd9ebf88dc92"}
Sep 11 13:19:50.183872 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.183841    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"dd9b10cf36455122293566cdc2a7ff9b5265590470f87bc256fccec891acfe51"}
Sep 11 13:19:50.183979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.183930    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"a6f4b68c7d2707938abfc43ba1ab20982d0bb88e878a165bfcd97df9a50cac3b"}
Sep 11 13:19:50.184041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.183982    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"6e10f005241050951af79bfc1b849dcdcde61a95675a55b4854e7061eedd787a"}
Sep 11 13:19:50.184396 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.184350    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-50.ec2.internal" podStartSLOduration=21.184333374 podStartE2EDuration="21.184333374s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:34.153327668 +0000 UTC m=+5.654232130" watchObservedRunningTime="2026-09-11 13:19:50.184333374 +0000 UTC m=+21.685237824"
Sep 11 13:19:50.184672 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.184594    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-image-registry/node-ca-9wzjp" podStartSLOduration=3.633130267 podStartE2EDuration="21.184580713s" totalImagesPullingTime="17.551450446s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:50.184113113 +0000 UTC m=+21.685017562" watchObservedRunningTime="2026-09-11 13:19:50.184580713 +0000 UTC m=+21.685485167"
Sep 11 13:19:50.185558 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.185532    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-6z4jd" event={"ID":"b863f06e-24b9-4893-80f4-d6923e658891","Type":"ContainerStarted","Data":"169ce792e41517b7c524501556f59e20a700308892719b70e7320f2081dc75cb"}
Sep 11 13:19:50.187357 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.187317    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-cf7rg" event={"ID":"b6b66705-ca0f-41f6-80fa-b6c4e478ccdb","Type":"ContainerStarted","Data":"c65899bfcc74bdbab73ea5e17512eaccf33c72a93b4209fc91f6a8c9fdad884b"}
Sep 11 13:19:50.201985 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.201951    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-pr4dg" podStartSLOduration=3.643098155 podStartE2EDuration="21.201939098s" totalImagesPullingTime="17.558840943s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:50.201620205 +0000 UTC m=+21.702524654" watchObservedRunningTime="2026-09-11 13:19:50.201939098 +0000 UTC m=+21.702843555"
Sep 11 13:19:50.225620 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.225576    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/node-resolver-gh9gl" podStartSLOduration=3.700821241 podStartE2EDuration="21.225562586s" totalImagesPullingTime="17.524741345s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:50.225197207 +0000 UTC m=+21.726101676" watchObservedRunningTime="2026-09-11 13:19:50.225562586 +0000 UTC m=+21.726467035"
Sep 11 13:19:50.319191 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.317836    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/konnectivity-agent-6z4jd" podStartSLOduration=11.025480107 podStartE2EDuration="21.317816024s" totalImagesPullingTime="10.292335917s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:50.317797694 +0000 UTC m=+21.818702170" watchObservedRunningTime="2026-09-11 13:19:50.317816024 +0000 UTC m=+21.818720487"
Sep 11 13:19:50.348078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.348036    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-cf7rg" podStartSLOduration=3.671781143 podStartE2EDuration="21.348014685s" totalImagesPullingTime="17.676233542s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:50.346809735 +0000 UTC m=+21.847714208" watchObservedRunningTime="2026-09-11 13:19:50.348014685 +0000 UTC m=+21.848919137"
Sep 11 13:19:50.697385 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.697316    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:50.697948 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.697928    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:50.958070 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.957989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:50.958261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.957989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:50.958261 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:50.958137    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:50.958261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:50.957989    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:50.958261 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:50.958186    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:50.958472 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:50.958261    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:51.191083 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.191037    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-w249f" event={"ID":"09ce8773-3f3d-4122-ab10-c33cb17bb152","Type":"ContainerStarted","Data":"b37d82d0ffb79a65662010bbd3de601b44abd832649bdba2f4076811f01a0585"}
Sep 11 13:19:51.194657 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.194613    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"edabd73f54627b34aaa059a26cf9bf5fa9ddc05b464347ca5af8e24e1f258e9a"}
Sep 11 13:19:51.194784 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.194714    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"457518e0175a7950e33926915949de3479070c43fde6f15d1988af8804a86178"}
Sep 11 13:19:51.194784 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.194768    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"605814639ae75f9b608e4c3719741b9458efc5e4dc23a88ef4d20225f80940b0"}
Sep 11 13:19:51.194871 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.194823    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:51.195407 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.195385    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-6z4jd"
Sep 11 13:19:51.253915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.253851    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-w249f" podStartSLOduration=4.70503866 podStartE2EDuration="22.253831771s" totalImagesPullingTime="17.548793111s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:51.223318811 +0000 UTC m=+22.724223299" watchObservedRunningTime="2026-09-11 13:19:51.253831771 +0000 UTC m=+22.754736219"
Sep 11 13:19:51.790510 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.790486    2551 plugin_watcher.go:194] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Sep 11 13:19:51.962769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.962579    2551 reconciler.go:161] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-09-11T13:19:51.790504103Z","UUID":"e38303e3-bfc7-4658-a534-92eb2e15f871","Handler":null,"Name":"","Endpoint":""}
Sep 11 13:19:51.964595 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.964431    2551 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Sep 11 13:19:51.964731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:51.964609    2551 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Sep 11 13:19:52.198672 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:52.198620    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r" event={"ID":"c9b50cf3-1019-4849-849b-ac93610a294e","Type":"ContainerStarted","Data":"8e12c6a029de0ad8988dc858f0e63938fe9f827dbba2f5fbef611e38d8da44cd"}
Sep 11 13:19:52.958181 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:52.958147    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:52.958181 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:52.958179    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:52.958432 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:52.958153    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:52.958432 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:52.958286    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:52.958432 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:52.958344    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:52.958615 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:52.958447    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:53.205257 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:53.205223    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"f5e9ef3b2f5eda40744fe235a78c0ecacdc0b27819bbf08f5b99166ce9380228"}
Sep 11 13:19:54.212733 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:54.212691    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r" event={"ID":"c9b50cf3-1019-4849-849b-ac93610a294e","Type":"ContainerStarted","Data":"dbf8b3327c2f3ece57062b23ef83ef62117e633f305a8ff63074a615a94b273a"}
Sep 11 13:19:54.957952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:54.957865    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:54.958124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:54.957865    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:54.958124 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:54.958007    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:54.958124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:54.958040    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:54.958276 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:54.958138    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:54.958276 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:54.958201    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:56.218485 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.218311    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="48300d098c42297e7b0a7987a260382f2b3dbdbfcce5498c2e74b2ccc7487d28" exitCode=0
Sep 11 13:19:56.218994 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.218365    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"48300d098c42297e7b0a7987a260382f2b3dbdbfcce5498c2e74b2ccc7487d28"}
Sep 11 13:19:56.222380 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.222356    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" event={"ID":"1657531f-635c-48c7-917d-30db46e0acfd","Type":"ContainerStarted","Data":"eba10273d2cbcafd985e23c7691bbee499abebc1af024dea22d90f3e9594b0a3"}
Sep 11 13:19:56.222619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.222600    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:56.222817 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.222681    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:56.222817 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.222726    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:56.236912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.236887    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:56.237025 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.236996    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:19:56.244678 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.244594    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-7zk2r" podStartSLOduration=6.682498617 podStartE2EDuration="27.244581778s" totalImagesPullingTime="20.562083161s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:54.245337731 +0000 UTC m=+25.746242181" watchObservedRunningTime="2026-09-11 13:19:56.244581778 +0000 UTC m=+27.745486226"
Sep 11 13:19:56.958731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.958701    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:56.959006 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:56.958826    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:56.959006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.958841    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:56.959006 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:56.958988    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:56.959188 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:56.959021    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:56.959188 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:56.959095    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:57.472418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.472361    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls" podStartSLOduration=10.89176267 podStartE2EDuration="28.472341673s" totalImagesPullingTime="17.580579003s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:19:56.281728703 +0000 UTC m=+27.782633163" watchObservedRunningTime="2026-09-11 13:19:57.472341673 +0000 UTC m=+28.973246119"
Sep 11 13:19:57.473427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.473403    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgb2x"]
Sep 11 13:19:57.473565 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.473552    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:57.473670 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:57.473657    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:57.477485 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.477460    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r9tc6"]
Sep 11 13:19:57.477615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.477601    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:57.477754 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:57.477728    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:19:57.478042 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.478023    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fm9hc"]
Sep 11 13:19:57.478126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:57.478115    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:57.478253 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:57.478227    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:58.228952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:58.228917    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="2662dff14ec9464c08a55d5c7950e46751c18baa8a873456379a44fd8afe9ec1" exitCode=0
Sep 11 13:19:58.229144 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:58.229003    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"2662dff14ec9464c08a55d5c7950e46751c18baa8a873456379a44fd8afe9ec1"}
Sep 11 13:19:58.959102 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:58.959070    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:19:58.959547 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:58.959121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:19:58.959547 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:19:58.959181    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:19:58.959547 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:58.959193    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:19:58.959547 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:58.959293    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:19:58.959547 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:19:58.959299    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:20:00.236414 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:00.236224    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="8196190ac9b20304e04a947c557c0c035c15f281175f2ccf11e863641efc6e39" exitCode=0
Sep 11 13:20:00.236885 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:00.236300    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"8196190ac9b20304e04a947c557c0c035c15f281175f2ccf11e863641efc6e39"}
Sep 11 13:20:00.957937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:00.957857    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:00.957937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:00.957884    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:20:00.958167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:00.957957    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:00.958167 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:00.958063    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-fm9hc" podUID="6143b386-05b7-46e3-8509-b55c72bc2c4d"
Sep 11 13:20:00.958167 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:00.958140    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-r9tc6" podUID="3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7"
Sep 11 13:20:00.958318 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:00.958224    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-zgb2x" podUID="06617998-5730-4981-9d6d-1ccbc62c7ab5"
Sep 11 13:20:02.648984 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.648892    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:20:02.649376 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.649031    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:20:02.649376 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.649095    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:34.649079637 +0000 UTC m=+66.149984069 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : object "openshift-multus"/"metrics-daemon-secret" not registered
Sep 11 13:20:02.749453 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.749417    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:02.749453 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.749464    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:02.749675 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749571    2551 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Sep 11 13:20:02.749675 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749656    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret podName:06617998-5730-4981-9d6d-1ccbc62c7ab5 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:34.749642244 +0000 UTC m=+66.250546684 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret") pod "global-pull-secret-syncer-zgb2x" (UID: "06617998-5730-4981-9d6d-1ccbc62c7ab5") : object "kube-system"/"original-pull-secret" not registered
Sep 11 13:20:02.749675 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749579    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Sep 11 13:20:02.749777 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749683    2551 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Sep 11 13:20:02.749777 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749693    2551 projected.go:196] Error preparing data for projected volume kube-api-access-596cv for pod openshift-network-diagnostics/network-check-target-fm9hc: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:20:02.749777 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:02.749737    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv podName:6143b386-05b7-46e3-8509-b55c72bc2c4d nodeName:}" failed. No retries permitted until 2026-09-11 13:20:34.749724013 +0000 UTC m=+66.250628439 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "kube-api-access-596cv" (UniqueName: "kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv") pod "network-check-target-fm9hc" (UID: "6143b386-05b7-46e3-8509-b55c72bc2c4d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Sep 11 13:20:02.862986 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.862948    2551 kubelet_node_status.go:695] "Recording event message for node" node="ip-10-0-141-50.ec2.internal" event="NodeReady"
Sep 11 13:20:02.863160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.863095    2551 kubelet_node_status.go:489] "Fast updating node status as it just became ready"
Sep 11 13:20:02.910023 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.909946    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-77ffb6df8-gn92c"]
Sep 11 13:20:02.912086 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.912060    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:02.912781 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.912461    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"]
Sep 11 13:20:02.914664 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.914643    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"]
Sep 11 13:20:02.914751 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.914723    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:02.916196 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.916177    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"]
Sep 11 13:20:02.916306 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.916291    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:02.917844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.917820    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"open-cluster-management-image-pull-credentials\""
Sep 11 13:20:02.918059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918039    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:02.918137 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918089    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"kube-root-ca.crt\""
Sep 11 13:20:02.918457 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918442    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-hub-kubeconfig\""
Sep 11 13:20:02.918457 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918448    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"open-cluster-management-agent-addon\"/\"openshift-service-ca.crt\""
Sep 11 13:20:02.918929 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918901    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"work-manager-hub-kubeconfig\""
Sep 11 13:20:02.919048 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.918933    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"managed-serviceaccount-dockercfg-76gc9\""
Sep 11 13:20:02.919246 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.919229    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Sep 11 13:20:02.919491 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.919473    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-g6rbf\""
Sep 11 13:20:02.919881 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.919862    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Sep 11 13:20:02.922711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.922661    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-ca\""
Sep 11 13:20:02.922711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.922661    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Sep 11 13:20:02.922865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.922822    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-open-cluster-management.io-proxy-agent-signer-client-cert\""
Sep 11 13:20:02.923112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.923093    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-service-proxy-server-certificates\""
Sep 11 13:20:02.923237 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.923217    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"open-cluster-management-agent-addon\"/\"cluster-proxy-hub-kubeconfig\""
Sep 11 13:20:02.929478 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.929457    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Sep 11 13:20:02.933339 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.933174    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-77ffb6df8-gn92c"]
Sep 11 13:20:02.937085 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.937016    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"]
Sep 11 13:20:02.937085 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.937067    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"]
Sep 11 13:20:02.941380 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.941351    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"]
Sep 11 13:20:02.949737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.949713    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-9f4s6"]
Sep 11 13:20:02.951585 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.951565    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-p7lsz"]
Sep 11 13:20:02.951750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.951712    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:02.953638 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.953608    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:02.957062 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.957045    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"default-dockercfg-rsfmf\""
Sep 11 13:20:02.957396 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.957381    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Sep 11 13:20:02.957446 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.957386    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Sep 11 13:20:02.957446 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.957425    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Sep 11 13:20:02.958448 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.958432    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:02.958516 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.958434    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:02.958574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.958541    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:20:02.960998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.960961    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-flkcx\""
Sep 11 13:20:02.961070 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.961018    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:20:02.961070 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.960965    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:20:02.961615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.961602    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sn4k5\""
Sep 11 13:20:02.962917 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.962900    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Sep 11 13:20:02.962917 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.962915    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:20:02.963065 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.962920    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:20:02.963065 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.962926    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Sep 11 13:20:02.967049 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.967034    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2xgs9\""
Sep 11 13:20:02.978717 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.977414    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p7lsz"]
Sep 11 13:20:02.979644 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:02.979601    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9f4s6"]
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.050859    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e394734-e060-40bc-b46a-1bc41dd4a4d2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.050900    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.050926    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4vthx\" (UniqueName: \"kubernetes.io/projected/df154d37-b493-494a-bd3c-6ec1a72b0132-kube-api-access-4vthx\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.050953    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.050979    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051019    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051039    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58cd463-a541-401b-99d7-9f976a8e6222-cert\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051060    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/df154d37-b493-494a-bd3c-6ec1a72b0132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051080    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051103    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051161    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqlsd\" (UniqueName: \"kubernetes.io/projected/d44b0dc9-4ee6-459e-98fb-a82a19841e55-kube-api-access-sqlsd\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051238    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77a3766-5474-48db-81c5-151c230e9936-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051365    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tc6bp\" (UniqueName: \"kubernetes.io/projected/2e394734-e060-40bc-b46a-1bc41dd4a4d2-kube-api-access-tc6bp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tchkq\" (UniqueName: \"kubernetes.io/projected/e77a3766-5474-48db-81c5-151c230e9936-kube-api-access-tchkq\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051410    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mh5dc\" (UniqueName: \"kubernetes.io/projected/e58cd463-a541-401b-99d7-9f976a8e6222-kube-api-access-mh5dc\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051428    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgzxb\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051493    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051524    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e394734-e060-40bc-b46a-1bc41dd4a4d2-tmp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051587    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44b0dc9-4ee6-459e-98fb-a82a19841e55-tmp-dir\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051615    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051655    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051684    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b0dc9-4ee6-459e-98fb-a82a19841e55-config-volume\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.051770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.051709    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152294    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tchkq\" (UniqueName: \"kubernetes.io/projected/e77a3766-5474-48db-81c5-151c230e9936-kube-api-access-tchkq\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152351    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mh5dc\" (UniqueName: \"kubernetes.io/projected/e58cd463-a541-401b-99d7-9f976a8e6222-kube-api-access-mh5dc\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152380    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgzxb\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152410    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152433    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152477    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e394734-e060-40bc-b46a-1bc41dd4a4d2-tmp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152501    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44b0dc9-4ee6-459e-98fb-a82a19841e55-tmp-dir\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.152535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152530    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152552    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152582    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b0dc9-4ee6-459e-98fb-a82a19841e55-config-volume\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152608    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152660    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e394734-e060-40bc-b46a-1bc41dd4a4d2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152684    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152709    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4vthx\" (UniqueName: \"kubernetes.io/projected/df154d37-b493-494a-bd3c-6ec1a72b0132-kube-api-access-4vthx\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152740    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152769    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152808    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152834    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58cd463-a541-401b-99d7-9f976a8e6222-cert\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152860    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/df154d37-b493-494a-bd3c-6ec1a72b0132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.152928 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152938    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/d44b0dc9-4ee6-459e-98fb-a82a19841e55-tmp-dir\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.152946    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqlsd\" (UniqueName: \"kubernetes.io/projected/d44b0dc9-4ee6-459e-98fb-a82a19841e55-kube-api-access-sqlsd\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153034    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77a3766-5474-48db-81c5-151c230e9936-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153066    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153126    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tc6bp\" (UniqueName: \"kubernetes.io/projected/2e394734-e060-40bc-b46a-1bc41dd4a4d2-kube-api-access-tc6bp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153180    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153203    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/d44b0dc9-4ee6-459e-98fb-a82a19841e55-config-volume\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.153295    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.153333    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/2e394734-e060-40bc-b46a-1bc41dd4a4d2-tmp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.153574 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.153355    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:03.653338594 +0000 UTC m=+35.154243022 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:03.154209 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.154184    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"additional-service-ca\" (UniqueName: \"kubernetes.io/configmap/df154d37-b493-494a-bd3c-6ec1a72b0132-additional-service-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.154583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.154399    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.155926 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.155899    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:03.155926 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.155926    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:03.156085 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.155981    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:03.655964391 +0000 UTC m=+35.156868837 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:03.156556 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.156287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.158273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.158204    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.158273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.158250    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"ca\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-ca\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.158419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.158295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.158787 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.158766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.159326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.159302    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/e58cd463-a541-401b-99d7-9f976a8e6222-cert\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.159723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.159672    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"service-proxy-server-cert\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-service-proxy-server-cert\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.160257 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.160203    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/e77a3766-5474-48db-81c5-151c230e9936-hub-kubeconfig\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.160359 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.160340    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"hub-kubeconfig\" (UniqueName: \"kubernetes.io/secret/df154d37-b493-494a-bd3c-6ec1a72b0132-hub-kubeconfig\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.160814 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.160751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"klusterlet-config\" (UniqueName: \"kubernetes.io/secret/2e394734-e060-40bc-b46a-1bc41dd4a4d2-klusterlet-config\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.174849 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.174794    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqlsd\" (UniqueName: \"kubernetes.io/projected/d44b0dc9-4ee6-459e-98fb-a82a19841e55-kube-api-access-sqlsd\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.175546 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.175523    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.176641 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.176606    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4vthx\" (UniqueName: \"kubernetes.io/projected/df154d37-b493-494a-bd3c-6ec1a72b0132-kube-api-access-4vthx\") pod \"cluster-proxy-proxy-agent-8484fd56fd-xkv6c\" (UID: \"df154d37-b493-494a-bd3c-6ec1a72b0132\") " pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.177022 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.177005    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tchkq\" (UniqueName: \"kubernetes.io/projected/e77a3766-5474-48db-81c5-151c230e9936-kube-api-access-tchkq\") pod \"managed-serviceaccount-addon-agent-59757f5474-jw9j7\" (UID: \"e77a3766-5474-48db-81c5-151c230e9936\") " pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.182701 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.182619    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgzxb\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.195929 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.195897    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tc6bp\" (UniqueName: \"kubernetes.io/projected/2e394734-e060-40bc-b46a-1bc41dd4a4d2-kube-api-access-tc6bp\") pod \"klusterlet-addon-workmgr-7657ff47d9-s2m78\" (UID: \"2e394734-e060-40bc-b46a-1bc41dd4a4d2\") " pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.196224 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.196205    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mh5dc\" (UniqueName: \"kubernetes.io/projected/e58cd463-a541-401b-99d7-9f976a8e6222-kube-api-access-mh5dc\") pod \"ingress-canary-9f4s6\" (UID: \"e58cd463-a541-401b-99d7-9f976a8e6222\") " pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.242131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.242105    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"
Sep 11 13:20:03.251118 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.251082    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:03.257157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.257124    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:20:03.275821 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.275569    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-9f4s6"
Sep 11 13:20:03.292300 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.292241    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:03.426090 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.425841    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:03.560330 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.560294    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"]
Sep 11 13:20:03.561192 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.561160    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"]
Sep 11 13:20:03.561848 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.561829    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7"]
Sep 11 13:20:03.587689 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.587657    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-9f4s6"]
Sep 11 13:20:03.657125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.657091    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:03.657125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:03.657126    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:03.657578 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.657270    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:03.657578 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.657282    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:03.657578 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.657336    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:04.657316491 +0000 UTC m=+36.158220940 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:03.657781 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.657755    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:03.657889 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:03.657832    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:04.657812401 +0000 UTC m=+36.158716851 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:04.666482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:04.666451    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:04.666482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:04.666486    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:04.667041 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:04.666608    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:04.667041 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:04.666645    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:04.667041 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:04.666651    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:04.667041 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:04.666716    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:06.66669765 +0000 UTC m=+38.167602092 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:04.667041 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:04.666733    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:06.666725026 +0000 UTC m=+38.167629458 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:04.906033 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:04.905987    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:05.011312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.011272    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:05.248772 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.248739    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerStarted","Data":"43c842f23a4eba45ee826f55e6e999f4ef83df476bfbb1e19146feda3a8f97c3"}
Sep 11 13:20:05.249811 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.249784    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9f4s6" event={"ID":"e58cd463-a541-401b-99d7-9f976a8e6222","Type":"ContainerStarted","Data":"ff7c1c8c41fe9c32cd4000b7d5608c824d333e28c912808b8879d0ad638aa932"}
Sep 11 13:20:05.250858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.250835    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7" event={"ID":"e77a3766-5474-48db-81c5-151c230e9936","Type":"ContainerStarted","Data":"f523cda1286cc16d7d70e68f4b5a95bfd44fafba8dc7397e3260d6a8f725905a"}
Sep 11 13:20:05.251946 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.251923    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78" event={"ID":"2e394734-e060-40bc-b46a-1bc41dd4a4d2","Type":"ContainerStarted","Data":"29cb02721c265d604d18654f488b452733129527246cd4c9e6f0386a4c43da18"}
Sep 11 13:20:05.254708 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.254686    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="ac509c6e0dcba8aaf59ce5f109f49772e6f825e184956c577720508b2f148309" exitCode=0
Sep 11 13:20:05.254784 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.254724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"ac509c6e0dcba8aaf59ce5f109f49772e6f825e184956c577720508b2f148309"}
Sep 11 13:20:05.751434 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.751397    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:05.875614 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.875574    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:05.967598 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:05.967501    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:06.274211 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.274170    2551 generic.go:356] "Generic (PLEG): container finished" podID="d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d" containerID="26446b8465926c81381db95bf122ee5d4e3399da1be3e8e4bb29a24697e6df46" exitCode=0
Sep 11 13:20:06.274393 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.274229    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerDied","Data":"26446b8465926c81381db95bf122ee5d4e3399da1be3e8e4bb29a24697e6df46"}
Sep 11 13:20:06.663071 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.662984    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.685313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.685357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:06.685577    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:06.685593    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:06.685670    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:10.685650048 +0000 UTC m=+42.186554479 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:06.686103    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:06.686190 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:06.686154    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:10.68613822 +0000 UTC m=+42.187042662 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:06.761779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:06.761725    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:07.285726 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:07.284724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-snvtd" event={"ID":"d8b69eb1-44c3-4ff1-8b28-b6e4d412c58d","Type":"ContainerStarted","Data":"138293f0e9be9e2201bbec43d9bbac4bdb27bc01559566cedc555c0d6b04eb2e"}
Sep 11 13:20:07.535905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:07.535794    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:07.661507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:07.661343    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:07.809008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:07.808923    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:07.914903 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:07.914859    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:08.701687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:08.701615    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:09.117353 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:09.117318    2551 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log" limit=1024
Sep 11 13:20:10.724716 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:10.724683    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:10.724716 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:10.724718    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:10.725219 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:10.724824    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:10.725219 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:10.724843    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:10.725219 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:10.724856    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:10.725219 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:10.724896    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:18.724878522 +0000 UTC m=+50.225782954 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:10.725219 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:10.724915    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:18.724906043 +0000 UTC m=+50.225810470 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:14.303842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.303806    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-9f4s6" event={"ID":"e58cd463-a541-401b-99d7-9f976a8e6222","Type":"ContainerStarted","Data":"1df3f21919852bf8fafaa11afabae59bda90119b94fdff12bbe747043c5b52c5"}
Sep 11 13:20:14.305476 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.305449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7" event={"ID":"e77a3766-5474-48db-81c5-151c230e9936","Type":"ContainerStarted","Data":"dd9be56d27c4fb44f4f39d7a1af23c47c482b7e8e56a58ca542049a8a1486119"}
Sep 11 13:20:14.307076 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.307043    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78" event={"ID":"2e394734-e060-40bc-b46a-1bc41dd4a4d2","Type":"ContainerStarted","Data":"f8104da14c78449f143a1c31cbb99c42a05fa275f9be39a10734b3c5b6739e97"}
Sep 11 13:20:14.307165 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.307105    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:14.309037 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.309013    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerStarted","Data":"4914fc51d0157b6049a003acefae55f88a8b5dfa3813c1f88ea616dd98d1dec5"}
Sep 11 13:20:14.309126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.309068    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78"
Sep 11 13:20:14.309126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.309087    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerStarted","Data":"f7920bc4da8bb7338acaf189baf681701b0f0d59694352f97267eed8ed22c490"}
Sep 11 13:20:14.323516 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.323480    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-snvtd" podStartSLOduration=22.781508755 podStartE2EDuration="45.323450764s" totalImagesPullingTime="22.541942009s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:07.317450326 +0000 UTC m=+38.818354789" watchObservedRunningTime="2026-09-11 13:20:14.323450764 +0000 UTC m=+45.824355194"
Sep 11 13:20:14.324280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.324246    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-9f4s6" podStartSLOduration=3.653522349 podStartE2EDuration="12.324232694s" totalImagesPullingTime="8.670710345s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:14.322923194 +0000 UTC m=+45.823827642" watchObservedRunningTime="2026-09-11 13:20:14.324232694 +0000 UTC m=+45.825137144"
Sep 11 13:20:14.349208 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.349154    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/managed-serviceaccount-addon-agent-59757f5474-jw9j7" podStartSLOduration=31.678618845 podStartE2EDuration="40.349132814s" totalImagesPullingTime="8.670513969s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:14.347016879 +0000 UTC m=+45.847921325" watchObservedRunningTime="2026-09-11 13:20:14.349132814 +0000 UTC m=+45.850037269"
Sep 11 13:20:14.380033 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:14.379984    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/klusterlet-addon-workmgr-7657ff47d9-s2m78" podStartSLOduration=31.692060513 podStartE2EDuration="40.379968006s" totalImagesPullingTime="8.687907493s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:14.37938259 +0000 UTC m=+45.880287056" watchObservedRunningTime="2026-09-11 13:20:14.379968006 +0000 UTC m=+45.880872456"
Sep 11 13:20:15.315099 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:15.314995    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerStarted","Data":"3c04fa652d90fed2fda52039051504c5c2207d1bc96061d355928ac6449a9ba0"}
Sep 11 13:20:15.338935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:15.338883    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" podStartSLOduration=32.266085016 podStartE2EDuration="41.338867087s" totalImagesPullingTime="9.072782071s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:15.338547905 +0000 UTC m=+46.839452354" watchObservedRunningTime="2026-09-11 13:20:15.338867087 +0000 UTC m=+46.839771533"
Sep 11 13:20:18.787188 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:18.787146    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:18.787188 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:18.787186    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:18.787590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:18.787298    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:18.787590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:18.787365    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:20:34.787351231 +0000 UTC m=+66.288255657 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:18.787590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:18.787302    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:18.787590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:18.787390    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:18.787590 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:18.787437    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:20:34.787427284 +0000 UTC m=+66.288331723 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:28.251941 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:28.251903    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-rqzls"
Sep 11 13:20:34.687128 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.687088    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:20:34.690548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.690524    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Sep 11 13:20:34.697452 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.697436    2551 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Sep 11 13:20:34.697518 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.697493    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs podName:3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:38.697476308 +0000 UTC m=+130.198380735 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs") pod "network-metrics-daemon-r9tc6" (UID: "3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7") : secret "metrics-daemon-secret" not found
Sep 11 13:20:34.787755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.787716    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:34.787755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.787762    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.787782    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.787799    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.787873    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.787879    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.787951    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:06.787931309 +0000 UTC m=+98.288835758 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:20:34.787991 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.787886    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:20:34.788289 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:20:34.788017    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:21:06.78800541 +0000 UTC m=+98.288909837 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:20:34.790582 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.790559    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Sep 11 13:20:34.790711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.790596    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Sep 11 13:20:34.799889 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.799873    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Sep 11 13:20:34.801057 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.801038    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/06617998-5730-4981-9d6d-1ccbc62c7ab5-original-pull-secret\") pod \"global-pull-secret-syncer-zgb2x\" (UID: \"06617998-5730-4981-9d6d-1ccbc62c7ab5\") " pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:34.810677 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:34.810654    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-596cv\" (UniqueName: \"kubernetes.io/projected/6143b386-05b7-46e3-8509-b55c72bc2c4d-kube-api-access-596cv\") pod \"network-check-target-fm9hc\" (UID: \"6143b386-05b7-46e3-8509-b55c72bc2c4d\") " pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:35.097234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:35.097206    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"default-dockercfg-sn4k5\""
Sep 11 13:20:35.101008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:35.100991    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-zgb2x"
Sep 11 13:20:35.105140 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:35.105121    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:35.394820 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:35.394751    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-zgb2x"]
Sep 11 13:20:35.397884 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:20:35.397858    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06617998_5730_4981_9d6d_1ccbc62c7ab5.slice/crio-0583dcab08a24d1052690e1adafd125cac413235415f590c5058504755a6b7ae WatchSource:0}: Error finding container 0583dcab08a24d1052690e1adafd125cac413235415f590c5058504755a6b7ae: Status 404 returned error can't find the container with id 0583dcab08a24d1052690e1adafd125cac413235415f590c5058504755a6b7ae
Sep 11 13:20:35.428978 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:35.428951    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-fm9hc"]
Sep 11 13:20:35.431897 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:20:35.431871    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6143b386_05b7_46e3_8509_b55c72bc2c4d.slice/crio-3e9d9140be9f271a085a170074015f3f199c4e702793a31c83b2fc0af2b65f6c WatchSource:0}: Error finding container 3e9d9140be9f271a085a170074015f3f199c4e702793a31c83b2fc0af2b65f6c: Status 404 returned error can't find the container with id 3e9d9140be9f271a085a170074015f3f199c4e702793a31c83b2fc0af2b65f6c
Sep 11 13:20:36.378723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:36.378675    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fm9hc" event={"ID":"6143b386-05b7-46e3-8509-b55c72bc2c4d","Type":"ContainerStarted","Data":"3e9d9140be9f271a085a170074015f3f199c4e702793a31c83b2fc0af2b65f6c"}
Sep 11 13:20:36.380598 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:36.380554    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgb2x" event={"ID":"06617998-5730-4981-9d6d-1ccbc62c7ab5","Type":"ContainerStarted","Data":"0583dcab08a24d1052690e1adafd125cac413235415f590c5058504755a6b7ae"}
Sep 11 13:20:41.398731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:41.398694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-fm9hc" event={"ID":"6143b386-05b7-46e3-8509-b55c72bc2c4d","Type":"ContainerStarted","Data":"77ef57a2d40ffcdf5a6a14cd653f42ada7bf1f26f3b6216ced728ad75a6c6893"}
Sep 11 13:20:41.399219 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:41.398823    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:20:41.400135 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:41.400103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-zgb2x" event={"ID":"06617998-5730-4981-9d6d-1ccbc62c7ab5","Type":"ContainerStarted","Data":"53be1198d4459aaa18d43feea0692e117d81c6af8ec0d8ff1092b7adc8eb016a"}
Sep 11 13:20:41.421708 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:41.421662    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-fm9hc" podStartSLOduration=67.453763596 podStartE2EDuration="1m12.421643828s" totalImagesPullingTime="4.967880232s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:41.421507284 +0000 UTC m=+72.922411734" watchObservedRunningTime="2026-09-11 13:20:41.421643828 +0000 UTC m=+72.922548274"
Sep 11 13:20:41.450831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:20:41.450789    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kube-system/global-pull-secret-syncer-zgb2x" podStartSLOduration=67.443945541 podStartE2EDuration="1m12.450771477s" totalImagesPullingTime="5.006825936s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:20:41.450731419 +0000 UTC m=+72.951635868" watchObservedRunningTime="2026-09-11 13:20:41.450771477 +0000 UTC m=+72.951675933"
Sep 11 13:21:06.824900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:06.824751    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:21:06.824900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:06.824789    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") pod \"image-registry-77ffb6df8-gn92c\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") " pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:21:06.824900 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:06.824897    2551 projected.go:266] Couldn't get secret openshift-image-registry/image-registry-tls: secret "image-registry-tls" not found
Sep 11 13:21:06.825383 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:06.824909    2551 projected.go:196] Error preparing data for projected volume registry-tls for pod openshift-image-registry/image-registry-77ffb6df8-gn92c: secret "image-registry-tls" not found
Sep 11 13:21:06.825383 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:06.824915    2551 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Sep 11 13:21:06.825383 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:06.824959    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls podName:f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa nodeName:}" failed. No retries permitted until 2026-09-11 13:22:10.824946078 +0000 UTC m=+162.325850505 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "registry-tls" (UniqueName: "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls") pod "image-registry-77ffb6df8-gn92c" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa") : secret "image-registry-tls" not found
Sep 11 13:21:06.825383 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:06.824986    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls podName:d44b0dc9-4ee6-459e-98fb-a82a19841e55 nodeName:}" failed. No retries permitted until 2026-09-11 13:22:10.82496569 +0000 UTC m=+162.325870130 (durationBeforeRetry 1m4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls") pod "dns-default-p7lsz" (UID: "d44b0dc9-4ee6-459e-98fb-a82a19841e55") : secret "dns-default-metrics-tls" not found
Sep 11 13:21:12.406014 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.405986    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-fm9hc"
Sep 11 13:21:12.723917 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.723227    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"]
Sep 11 13:21:12.727068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.727045    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"
Sep 11 13:21:12.756257 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.756224    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-diagnostics-dockercfg-8s2xw\""
Sep 11 13:21:12.836099 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.836070    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"]
Sep 11 13:21:12.870510 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.870477    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t4f44\" (UniqueName: \"kubernetes.io/projected/663d675b-d4cd-4601-a594-bbebfbcc00db-kube-api-access-t4f44\") pod \"network-check-source-6f8b768455-jtrvr\" (UID: \"663d675b-d4cd-4601-a594-bbebfbcc00db\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"
Sep 11 13:21:12.971069 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.971027    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t4f44\" (UniqueName: \"kubernetes.io/projected/663d675b-d4cd-4601-a594-bbebfbcc00db-kube-api-access-t4f44\") pod \"network-check-source-6f8b768455-jtrvr\" (UID: \"663d675b-d4cd-4601-a594-bbebfbcc00db\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"
Sep 11 13:21:12.980978 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:12.980908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-t4f44\" (UniqueName: \"kubernetes.io/projected/663d675b-d4cd-4601-a594-bbebfbcc00db-kube-api-access-t4f44\") pod \"network-check-source-6f8b768455-jtrvr\" (UID: \"663d675b-d4cd-4601-a594-bbebfbcc00db\") " pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"
Sep 11 13:21:13.034756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:13.034726    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"
Sep 11 13:21:13.315367 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:13.315290    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr"]
Sep 11 13:21:13.318308 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:21:13.318279    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod663d675b_d4cd_4601_a594_bbebfbcc00db.slice/crio-90625f27f28e8d96b04c25513f4c9802100011f000b68550459483a01769f579 WatchSource:0}: Error finding container 90625f27f28e8d96b04c25513f4c9802100011f000b68550459483a01769f579: Status 404 returned error can't find the container with id 90625f27f28e8d96b04c25513f4c9802100011f000b68550459483a01769f579
Sep 11 13:21:13.495893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:13.495854    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr" event={"ID":"663d675b-d4cd-4601-a594-bbebfbcc00db","Type":"ContainerStarted","Data":"10b71e2adab90b4e96c2ebaf9d192974e71d08508863d8a92c220ddc70cbce3c"}
Sep 11 13:21:13.496254 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:13.495918    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr" event={"ID":"663d675b-d4cd-4601-a594-bbebfbcc00db","Type":"ContainerStarted","Data":"90625f27f28e8d96b04c25513f4c9802100011f000b68550459483a01769f579"}
Sep 11 13:21:16.838186 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.837674    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-source-6f8b768455-jtrvr" podStartSLOduration=4.837649209 podStartE2EDuration="4.837649209s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:21:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:13.513092684 +0000 UTC m=+105.013997134" watchObservedRunningTime="2026-09-11 13:21:16.837649209 +0000 UTC m=+108.338553657"
Sep 11 13:21:16.838186 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.838079    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-hvnhq"]
Sep 11 13:21:16.840420 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.840404    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.849179 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.849158    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Sep 11 13:21:16.849296 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.849158    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Sep 11 13:21:16.849296 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.849199    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Sep 11 13:21:16.849377 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.849337    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Sep 11 13:21:16.849500 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.849477    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ndm9r\""
Sep 11 13:21:16.896758 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.896729    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7d7905c-8a44-41c9-9e66-9ce00ac29496-data-volume\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.896758 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.896762    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.896932 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.896784    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lf2m4\" (UniqueName: \"kubernetes.io/projected/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-api-access-lf2m4\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.896932 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.896894    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7d7905c-8a44-41c9-9e66-9ce00ac29496-crio-socket\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.897003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.896962    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.903753 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.903726    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvnhq"]
Sep 11 13:21:16.997695 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997659    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997695 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997695    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7d7905c-8a44-41c9-9e66-9ce00ac29496-data-volume\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997712    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997732    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lf2m4\" (UniqueName: \"kubernetes.io/projected/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-api-access-lf2m4\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997767    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7d7905c-8a44-41c9-9e66-9ce00ac29496-crio-socket\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:16.997788    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:16.997828    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.997850    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/a7d7905c-8a44-41c9-9e66-9ce00ac29496-crio-socket\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:16.997866    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:17.497845577 +0000 UTC m=+108.998750021 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : configmap "kube-rbac-proxy" not found
Sep 11 13:21:16.997898 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:16.997892    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:17.497876097 +0000 UTC m=+108.998780525 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:16.998156 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:16.998026    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/a7d7905c-8a44-41c9-9e66-9ce00ac29496-data-volume\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:17.007380 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:17.007354    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-lf2m4\" (UniqueName: \"kubernetes.io/projected/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-api-access-lf2m4\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:17.502670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:17.502609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:17.502670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:17.502673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:17.502918 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:17.502739    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:21:17.502918 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:17.502793    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:17.502918 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:17.502830    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:18.502810242 +0000 UTC m=+110.003714689 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : configmap "kube-rbac-proxy" not found
Sep 11 13:21:17.502918 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:17.502845    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:18.502838725 +0000 UTC m=+110.003743152 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:18.510427    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:18.510487    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:18.510722    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:18.510794    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:20.510768698 +0000 UTC m=+112.011673128 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:18.511244    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:21:18.512974 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:18.511301    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:20.511285676 +0000 UTC m=+112.012190125 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : configmap "kube-rbac-proxy" not found
Sep 11 13:21:18.834401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:18.834329    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gh9gl_817a83e4-76b4-4ac1-8afe-27c3beed2530/dns-node-resolver/0.log"
Sep 11 13:21:19.447188 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:19.447162    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wzjp_ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c/node-ca/0.log"
Sep 11 13:21:20.233851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:20.233813    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9f4s6_e58cd463-a541-401b-99d7-9f976a8e6222/serve-healthcheck-canary/0.log"
Sep 11 13:21:20.527416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:20.527329    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:20.527416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:20.527362    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:20.527694 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:20.527465    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:21:20.527694 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:20.527476    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:20.527694 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:20.527545    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:24.527524452 +0000 UTC m=+116.028428893 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : configmap "kube-rbac-proxy" not found
Sep 11 13:21:20.527694 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:20.527601    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:24.527589986 +0000 UTC m=+116.028494413 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:24.561714 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:24.561673    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:24.561714 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:24.561715    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:24.562233 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:24.561810    2551 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:24.562233 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:24.561812    2551 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Sep 11 13:21:24.562233 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:24.561895    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:32.561875336 +0000 UTC m=+124.062779772 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : configmap "kube-rbac-proxy" not found
Sep 11 13:21:24.562233 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:21:24.561913    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls podName:a7d7905c-8a44-41c9-9e66-9ce00ac29496 nodeName:}" failed. No retries permitted until 2026-09-11 13:21:32.561904834 +0000 UTC m=+124.062809260 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls") pod "insights-runtime-extractor-hvnhq" (UID: "a7d7905c-8a44-41c9-9e66-9ce00ac29496") : secret "insights-runtime-extractor-tls" not found
Sep 11 13:21:32.627597 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.627538    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:32.627597 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.627600    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:32.628083 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.628063    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/a7d7905c-8a44-41c9-9e66-9ce00ac29496-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:32.629862 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.629843    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/a7d7905c-8a44-41c9-9e66-9ce00ac29496-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-hvnhq\" (UID: \"a7d7905c-8a44-41c9-9e66-9ce00ac29496\") " pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:32.752079 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.752046    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-ndm9r\""
Sep 11 13:21:32.760014 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:32.759994    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-hvnhq"
Sep 11 13:21:33.034852 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:33.034817    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-hvnhq"]
Sep 11 13:21:33.037863 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:21:33.037831    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda7d7905c_8a44_41c9_9e66_9ce00ac29496.slice/crio-9d0399569e54abe19dd27c82b7fbae5cceeb10c7538543aa2e68eef2f8639437 WatchSource:0}: Error finding container 9d0399569e54abe19dd27c82b7fbae5cceeb10c7538543aa2e68eef2f8639437: Status 404 returned error can't find the container with id 9d0399569e54abe19dd27c82b7fbae5cceeb10c7538543aa2e68eef2f8639437
Sep 11 13:21:33.554415 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:33.554378    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvnhq" event={"ID":"a7d7905c-8a44-41c9-9e66-9ce00ac29496","Type":"ContainerStarted","Data":"170893352ff797120192154a26151e793feabe14ab2172a9231cb6cc453923c3"}
Sep 11 13:21:33.554580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:33.554443    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvnhq" event={"ID":"a7d7905c-8a44-41c9-9e66-9ce00ac29496","Type":"ContainerStarted","Data":"9d0399569e54abe19dd27c82b7fbae5cceeb10c7538543aa2e68eef2f8639437"}
Sep 11 13:21:34.560044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:34.560012    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvnhq" event={"ID":"a7d7905c-8a44-41c9-9e66-9ce00ac29496","Type":"ContainerStarted","Data":"02a019d4f02ba2c13cd91c5bf28dc77813dda34b1bb1af3c88f9eb6af0e42c02"}
Sep 11 13:21:35.564696 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:35.564657    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-hvnhq" event={"ID":"a7d7905c-8a44-41c9-9e66-9ce00ac29496","Type":"ContainerStarted","Data":"e00af93f4d58a81a0658802094c9a7bf24d68afd8274579ad9b4a88c9a1d064b"}
Sep 11 13:21:38.771794 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:38.771750    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:21:38.774001 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:38.773980    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7-metrics-certs\") pod \"network-metrics-daemon-r9tc6\" (UID: \"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7\") " pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:21:39.009919 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:39.009892    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-2xgs9\""
Sep 11 13:21:39.018691 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:39.018669    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-r9tc6"
Sep 11 13:21:39.283525 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:39.283417    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-insights/insights-runtime-extractor-hvnhq" podStartSLOduration=21.275755224 podStartE2EDuration="23.283397306s" totalImagesPullingTime="2.007642082s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:16 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:35.592370989 +0000 UTC m=+127.093275437" watchObservedRunningTime="2026-09-11 13:21:39.283397306 +0000 UTC m=+130.784301755"
Sep 11 13:21:39.283911 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:39.283886    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-r9tc6"]
Sep 11 13:21:39.288336 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:21:39.288314    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3fa4e1b6_07fe_4290_8e97_5dd0ddab5ce7.slice/crio-7bbdd7d137fce741a6d3caa41aa9610ecb828530ce28883b0ab24c53efc46f30 WatchSource:0}: Error finding container 7bbdd7d137fce741a6d3caa41aa9610ecb828530ce28883b0ab24c53efc46f30: Status 404 returned error can't find the container with id 7bbdd7d137fce741a6d3caa41aa9610ecb828530ce28883b0ab24c53efc46f30
Sep 11 13:21:39.577487 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:39.577404    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9tc6" event={"ID":"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7","Type":"ContainerStarted","Data":"7bbdd7d137fce741a6d3caa41aa9610ecb828530ce28883b0ab24c53efc46f30"}
Sep 11 13:21:40.582297 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:40.582266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9tc6" event={"ID":"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7","Type":"ContainerStarted","Data":"bdf7551df78773b3d5a899a7061af703143508ca4b0163b7d338ef114fee328e"}
Sep 11 13:21:41.587268 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:41.587231    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-r9tc6" event={"ID":"3fa4e1b6-07fe-4290-8e97-5dd0ddab5ce7","Type":"ContainerStarted","Data":"e0d20b6e312a256225b90d2a8e7cacd9caf3bec77fa7fbf45d01d44785a3739e"}
Sep 11 13:21:41.639255 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:41.639200    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-multus/network-metrics-daemon-r9tc6" podStartSLOduration=131.48345924 podStartE2EDuration="2m12.639184141s" totalImagesPullingTime="1.155724901s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:19:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:41.638115963 +0000 UTC m=+133.139020413" watchObservedRunningTime="2026-09-11 13:21:41.639184141 +0000 UTC m=+133.140088638"
Sep 11 13:21:53.258585 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:53.258539    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" podUID="df154d37-b493-494a-bd3c-6ec1a72b0132" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:21:56.592934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.592900    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-9pbd8"]
Sep 11 13:21:56.594902 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.594884    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600555 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600531    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-tls\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600694 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600583    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-textfile\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600742 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600715    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600790 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600777    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-root\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600805    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bnb57\" (UniqueName: \"kubernetes.io/projected/61d61321-ce79-40ab-b0ef-232eeadb4c3c-kube-api-access-bnb57\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600838    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-wtmp\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-metrics-client-ca\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.600905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600889    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-sys\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.601364 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.600930    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.603185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.603165    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Sep 11 13:21:56.603943 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.603924    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Sep 11 13:21:56.604052 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.603944    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Sep 11 13:21:56.604052 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.603973    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-t4pnf\""
Sep 11 13:21:56.604052 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.603993    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Sep 11 13:21:56.604052 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.604032    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Sep 11 13:21:56.607452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.607436    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Sep 11 13:21:56.701542 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701508    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-tls\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701571    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-textfile\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701609    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701667    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-root\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701690    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bnb57\" (UniqueName: \"kubernetes.io/projected/61d61321-ce79-40ab-b0ef-232eeadb4c3c-kube-api-access-bnb57\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-wtmp\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.701754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701733    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-metrics-client-ca\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701758    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-sys\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701779    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-root\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701813    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-sys\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701846    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701893    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-wtmp\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.701986    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-textfile\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702363 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.702292    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-metrics-client-ca\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.702410 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.702380    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-accelerators-collector-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.703984 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.703963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-tls\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.704107 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.704084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/61d61321-ce79-40ab-b0ef-232eeadb4c3c-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.715782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.715762    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-bnb57\" (UniqueName: \"kubernetes.io/projected/61d61321-ce79-40ab-b0ef-232eeadb4c3c-kube-api-access-bnb57\") pod \"node-exporter-9pbd8\" (UID: \"61d61321-ce79-40ab-b0ef-232eeadb4c3c\") " pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.903408 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:56.903317    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-9pbd8"
Sep 11 13:21:56.912108 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:21:56.912081    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod61d61321_ce79_40ab_b0ef_232eeadb4c3c.slice/crio-d04343c88459dc8a2e25747760b30978422d3115a3e559d20640340a0e539935 WatchSource:0}: Error finding container d04343c88459dc8a2e25747760b30978422d3115a3e559d20640340a0e539935: Status 404 returned error can't find the container with id d04343c88459dc8a2e25747760b30978422d3115a3e559d20640340a0e539935
Sep 11 13:21:57.639495 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:57.639454    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pbd8" event={"ID":"61d61321-ce79-40ab-b0ef-232eeadb4c3c","Type":"ContainerStarted","Data":"d04343c88459dc8a2e25747760b30978422d3115a3e559d20640340a0e539935"}
Sep 11 13:21:58.644341 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:58.644304    2551 generic.go:356] "Generic (PLEG): container finished" podID="61d61321-ce79-40ab-b0ef-232eeadb4c3c" containerID="67d88b2868c74cf7198cfacb233d7dec9c2980a87865a0aee0f0442e8651adec" exitCode=0
Sep 11 13:21:58.644762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:58.644364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pbd8" event={"ID":"61d61321-ce79-40ab-b0ef-232eeadb4c3c","Type":"ContainerDied","Data":"67d88b2868c74cf7198cfacb233d7dec9c2980a87865a0aee0f0442e8651adec"}
Sep 11 13:21:59.650335 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:59.650298    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pbd8" event={"ID":"61d61321-ce79-40ab-b0ef-232eeadb4c3c","Type":"ContainerStarted","Data":"9f18096a303eb94d7dee9b1d624e2dfc958c44a4d47627e32e062377c5df3eda"}
Sep 11 13:21:59.650739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:59.650349    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-9pbd8" event={"ID":"61d61321-ce79-40ab-b0ef-232eeadb4c3c","Type":"ContainerStarted","Data":"4d242c486cf2de0e25dded594ecdc2f74eea9aeb356e14568f82b171440778cb"}
Sep 11 13:21:59.752562 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:21:59.752511    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-9pbd8" podStartSLOduration=2.705644334 podStartE2EDuration="3.752488899s" totalImagesPullingTime="983.897559ms" totalInitContainerRuntime="62.947006ms" isStatefulPod=true podCreationTimestamp="2026-09-11 13:21:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:21:59.746430485 +0000 UTC m=+151.247334933" watchObservedRunningTime="2026-09-11 13:21:59.752488899 +0000 UTC m=+151.253393349"
Sep 11 13:22:03.258214 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.258171    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" podUID="df154d37-b493-494a-bd3c-6ec1a72b0132" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:22:03.379907 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.379869    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77ffb6df8-gn92c"]
Sep 11 13:22:03.380095 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:22:03.380078    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[registry-tls], unattached volumes=[], failed to process volumes=[]: context canceled" pod="openshift-image-registry/image-registry-77ffb6df8-gn92c" podUID="f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"
Sep 11 13:22:03.661609 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.661530    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:22:03.666989 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.666967    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:22:03.756982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.756947    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757001    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757031    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757052    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757071    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757104    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757145 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757138    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tgzxb\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb\") pod \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\" (UID: \"f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa\") "
Sep 11 13:22:03.757443 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757314    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted" (OuterVolumeSpecName: "ca-trust-extracted") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "ca-trust-extracted". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:22:03.757443 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757425    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca" (OuterVolumeSpecName: "trusted-ca") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "trusted-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:22:03.757548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.757446    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates" (OuterVolumeSpecName: "registry-certificates") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "registry-certificates". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:22:03.759484 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.759457    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb" (OuterVolumeSpecName: "kube-api-access-tgzxb") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "kube-api-access-tgzxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:22:03.759612 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.759491    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets" (OuterVolumeSpecName: "installation-pull-secrets") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "installation-pull-secrets". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:22:03.759612 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.759512    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token" (OuterVolumeSpecName: "bound-sa-token") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "bound-sa-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:22:03.759612 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.759556    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration" (OuterVolumeSpecName: "image-registry-private-configuration") pod "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" (UID: "f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa"). InnerVolumeSpecName "image-registry-private-configuration". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:22:03.858480 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858442    2551 reconciler_common.go:299] "Volume detached for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-ca-trust-extracted\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858480 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858472    2551 reconciler_common.go:299] "Volume detached for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-certificates\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858480 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858485    2551 reconciler_common.go:299] "Volume detached for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-image-registry-private-configuration\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858738 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858495    2551 reconciler_common.go:299] "Volume detached for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-bound-sa-token\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858738 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858504    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tgzxb\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-kube-api-access-tgzxb\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858738 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858513    2551 reconciler_common.go:299] "Volume detached for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-trusted-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:03.858738 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:03.858521    2551 reconciler_common.go:299] "Volume detached for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-installation-pull-secrets\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:04.664589 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:04.664556    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-77ffb6df8-gn92c"
Sep 11 13:22:05.200366 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:05.200330    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-image-registry/image-registry-77ffb6df8-gn92c"]
Sep 11 13:22:05.229986 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:05.229950    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-image-registry/image-registry-77ffb6df8-gn92c"]
Sep 11 13:22:05.269522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:05.269495    2551 reconciler_common.go:299] "Volume detached for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa-registry-tls\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:22:05.987146 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:22:05.987105    2551 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[metrics-tls], unattached volumes=[], failed to process volumes=[]: context deadline exceeded" pod="openshift-dns/dns-default-p7lsz" podUID="d44b0dc9-4ee6-459e-98fb-a82a19841e55"
Sep 11 13:22:06.670236 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:06.670199    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:06.962670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:06.962574    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa" path="/var/lib/kubelet/pods/f74d8d81-b2ea-4c3d-87a9-551d8af1aeaa/volumes"
Sep 11 13:22:10.913411 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:10.913351    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:10.915655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:10.915620    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/d44b0dc9-4ee6-459e-98fb-a82a19841e55-metrics-tls\") pod \"dns-default-p7lsz\" (UID: \"d44b0dc9-4ee6-459e-98fb-a82a19841e55\") " pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:11.177021 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:11.176931    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-flkcx\""
Sep 11 13:22:11.182274 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:11.182251    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:11.501463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:11.501435    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-p7lsz"]
Sep 11 13:22:11.504941 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:22:11.504916    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd44b0dc9_4ee6_459e_98fb_a82a19841e55.slice/crio-e5bb6948efa65ce8374233d4dbd7dbb58cf9d7473340115dca77a4a75d18dddc WatchSource:0}: Error finding container e5bb6948efa65ce8374233d4dbd7dbb58cf9d7473340115dca77a4a75d18dddc: Status 404 returned error can't find the container with id e5bb6948efa65ce8374233d4dbd7dbb58cf9d7473340115dca77a4a75d18dddc
Sep 11 13:22:11.686399 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:11.686364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7lsz" event={"ID":"d44b0dc9-4ee6-459e-98fb-a82a19841e55","Type":"ContainerStarted","Data":"e5bb6948efa65ce8374233d4dbd7dbb58cf9d7473340115dca77a4a75d18dddc"}
Sep 11 13:22:13.257958 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.257924    2551 prober.go:120] "Probe failed" probeType="Liveness" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" podUID="df154d37-b493-494a-bd3c-6ec1a72b0132" containerName="service-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Sep 11 13:22:13.258395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.257989    2551 kubelet.go:2654] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c"
Sep 11 13:22:13.258477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.258446    2551 kuberuntime_manager.go:1192] "Message for Container of pod" containerName="service-proxy" containerStatusID={"Type":"cri-o","ID":"3c04fa652d90fed2fda52039051504c5c2207d1bc96061d355928ac6449a9ba0"} pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" containerMessage="Container service-proxy failed liveness probe, will be restarted"
Sep 11 13:22:13.258517 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.258498    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" podUID="df154d37-b493-494a-bd3c-6ec1a72b0132" containerName="service-proxy" containerID="cri-o://3c04fa652d90fed2fda52039051504c5c2207d1bc96061d355928ac6449a9ba0" gracePeriod=30
Sep 11 13:22:13.695075 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.695041    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7lsz" event={"ID":"d44b0dc9-4ee6-459e-98fb-a82a19841e55","Type":"ContainerStarted","Data":"49e2a0f210107d28ff490ca611fb0f97331d716d1f229d6a1b8032eccd01354f"}
Sep 11 13:22:13.695283 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.695101    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-p7lsz" event={"ID":"d44b0dc9-4ee6-459e-98fb-a82a19841e55","Type":"ContainerStarted","Data":"e5cae550a91ba2cf483f74a5e6e4db0aa005df847242054b1bd2409bf100e2c2"}
Sep 11 13:22:13.695283 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.695133    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:13.697327 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.697302    2551 generic.go:356] "Generic (PLEG): container finished" podID="df154d37-b493-494a-bd3c-6ec1a72b0132" containerID="3c04fa652d90fed2fda52039051504c5c2207d1bc96061d355928ac6449a9ba0" exitCode=2
Sep 11 13:22:13.697428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.697358    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerDied","Data":"3c04fa652d90fed2fda52039051504c5c2207d1bc96061d355928ac6449a9ba0"}
Sep 11 13:22:13.697428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.697391    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="open-cluster-management-agent-addon/cluster-proxy-proxy-agent-8484fd56fd-xkv6c" event={"ID":"df154d37-b493-494a-bd3c-6ec1a72b0132","Type":"ContainerStarted","Data":"f3c0a9f5332b8d9076756cc2888be8931484ac62fe196a16dc998fe4bb04aef7"}
Sep 11 13:22:13.714213 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:13.714158    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-dns/dns-default-p7lsz" podStartSLOduration=129.90929967 podStartE2EDuration="2m11.714143384s" totalImagesPullingTime="1.804843714s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:20:02 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:22:13.713195001 +0000 UTC m=+165.214099451" watchObservedRunningTime="2026-09-11 13:22:13.714143384 +0000 UTC m=+165.215047833"
Sep 11 13:22:23.703669 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:23.703578    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-p7lsz"
Sep 11 13:22:32.218554 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:32.218520    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7lsz_d44b0dc9-4ee6-459e-98fb-a82a19841e55/dns/0.log"
Sep 11 13:22:32.227782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:32.227761    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7lsz_d44b0dc9-4ee6-459e-98fb-a82a19841e55/kube-rbac-proxy/0.log"
Sep 11 13:22:32.774426 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:22:32.774399    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gh9gl_817a83e4-76b4-4ac1-8afe-27c3beed2530/dns-node-resolver/0.log"
Sep 11 13:24:28.907176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:24:28.907151    2551 kubelet.go:1648] "Image garbage collection succeeded"
Sep 11 13:26:44.278230 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.278165    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q4sp7"]
Sep 11 13:26:44.280094 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.280078    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.282212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.282187    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Sep 11 13:26:44.282339 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.282210    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Sep 11 13:26:44.282851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.282835    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-r9xpf\""
Sep 11 13:26:44.291142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.291119    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q4sp7"]
Sep 11 13:26:44.365669 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.365607    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-bound-sa-token\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.365832 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.365682    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wd8w2\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-kube-api-access-wd8w2\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.466844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.466812    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-bound-sa-token\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.466975 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.466855    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wd8w2\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-kube-api-access-wd8w2\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.475245 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.475217    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-bound-sa-token\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.475375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.475350    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wd8w2\" (UniqueName: \"kubernetes.io/projected/b97edfba-1f4d-4cbc-8201-ee537dca304a-kube-api-access-wd8w2\") pod \"cert-manager-55b4d954d5-q4sp7\" (UID: \"b97edfba-1f4d-4cbc-8201-ee537dca304a\") " pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.588591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.588502    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-55b4d954d5-q4sp7"
Sep 11 13:26:44.855955 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.855884    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-55b4d954d5-q4sp7"]
Sep 11 13:26:44.859140 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:26:44.859108    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podb97edfba_1f4d_4cbc_8201_ee537dca304a.slice/crio-7e3b814c50d86ce05c527aa989bc4e768d933bc7f8fc82d258737d7cee015166 WatchSource:0}: Error finding container 7e3b814c50d86ce05c527aa989bc4e768d933bc7f8fc82d258737d7cee015166: Status 404 returned error can't find the container with id 7e3b814c50d86ce05c527aa989bc4e768d933bc7f8fc82d258737d7cee015166
Sep 11 13:26:44.860963 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:44.860947    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:26:45.564729 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:45.564690    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q4sp7" event={"ID":"b97edfba-1f4d-4cbc-8201-ee537dca304a","Type":"ContainerStarted","Data":"7e3b814c50d86ce05c527aa989bc4e768d933bc7f8fc82d258737d7cee015166"}
Sep 11 13:26:48.576787 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:48.576753    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-55b4d954d5-q4sp7" event={"ID":"b97edfba-1f4d-4cbc-8201-ee537dca304a","Type":"ContainerStarted","Data":"90574eb3d80a281cc40dc81ff4cc6e941150838722a6b703ae33150786108130"}
Sep 11 13:26:48.600670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:26:48.600596    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="cert-manager/cert-manager-55b4d954d5-q4sp7" podStartSLOduration=1.6645628719999999 podStartE2EDuration="4.600575985s" totalImagesPullingTime="2.936013113s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:26:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:26:48.599755872 +0000 UTC m=+440.100660312" watchObservedRunningTime="2026-09-11 13:26:48.600575985 +0000 UTC m=+440.101480435"
Sep 11 13:27:17.995449 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:17.995416    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"]
Sep 11 13:27:17.997675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:17.997654    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.000099 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.000075    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Sep 11 13:27:18.000229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.000113    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-dockercfg-dlvws\""
Sep 11 13:27:18.000229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.000075    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-manager-service-cert\""
Sep 11 13:27:18.000229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.000171    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"opendatahub-operator-controller-webhook-cert\""
Sep 11 13:27:18.000229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.000193    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Sep 11 13:27:18.029599 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.029568    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"]
Sep 11 13:27:18.098349 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.098313    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.098512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.098354    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-tmp\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.098512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.098379    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-manifests\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.098512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.098437    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.098615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.098519    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wm275\" (UniqueName: \"kubernetes.io/projected/56a7e15d-c3f0-4460-be97-ee7dc88886a9-kube-api-access-wm275\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.199649 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.199593    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wm275\" (UniqueName: \"kubernetes.io/projected/56a7e15d-c3f0-4460-be97-ee7dc88886a9-kube-api-access-wm275\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.199802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.199680    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.199802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.199713    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-tmp\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.199802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.199748    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-manifests\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.199908 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.199867    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.200148 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.200128    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-manifests\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.202015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.201989    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56a7e15d-c3f0-4460-be97-ee7dc88886a9-tmp\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.202168 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.202092    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-webhook-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.202290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.202270    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/56a7e15d-c3f0-4460-be97-ee7dc88886a9-apiservice-cert\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.209720 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.209699    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wm275\" (UniqueName: \"kubernetes.io/projected/56a7e15d-c3f0-4460-be97-ee7dc88886a9-kube-api-access-wm275\") pod \"opendatahub-operator-controller-manager-64959b9bbc-flljd\" (UID: \"56a7e15d-c3f0-4460-be97-ee7dc88886a9\") " pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.307623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.307543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:18.598756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.598693    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"]
Sep 11 13:27:18.601431 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:27:18.601406    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56a7e15d_c3f0_4460_be97_ee7dc88886a9.slice/crio-e1d204011068915a4971de5a38a9a4e752de18704960da523bd57630a33c0900 WatchSource:0}: Error finding container e1d204011068915a4971de5a38a9a4e752de18704960da523bd57630a33c0900: Status 404 returned error can't find the container with id e1d204011068915a4971de5a38a9a4e752de18704960da523bd57630a33c0900
Sep 11 13:27:18.676993 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:18.676955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd" event={"ID":"56a7e15d-c3f0-4460-be97-ee7dc88886a9","Type":"ContainerStarted","Data":"e1d204011068915a4971de5a38a9a4e752de18704960da523bd57630a33c0900"}
Sep 11 13:27:22.694135 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:22.694094    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd" event={"ID":"56a7e15d-c3f0-4460-be97-ee7dc88886a9","Type":"ContainerStarted","Data":"d6ed272a1f62744e439138ffeb19120f12f2ae798545a62aa61f11da38655b4a"}
Sep 11 13:27:23.698991 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:23.698953    2551 generic.go:356] "Generic (PLEG): container finished" podID="56a7e15d-c3f0-4460-be97-ee7dc88886a9" containerID="d6ed272a1f62744e439138ffeb19120f12f2ae798545a62aa61f11da38655b4a" exitCode=0
Sep 11 13:27:23.699365 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:23.698999    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd" event={"ID":"56a7e15d-c3f0-4460-be97-ee7dc88886a9","Type":"ContainerDied","Data":"d6ed272a1f62744e439138ffeb19120f12f2ae798545a62aa61f11da38655b4a"}
Sep 11 13:27:24.704888 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:24.704858    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd" event={"ID":"56a7e15d-c3f0-4460-be97-ee7dc88886a9","Type":"ContainerStarted","Data":"4f3ac5954058d73417fdbb42d7b01088e37c9cd35a396a741feb1a158bf12fae"}
Sep 11 13:27:24.705258 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:24.705012    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:24.729324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:24.729274    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd" podStartSLOduration=3.24592722 podStartE2EDuration="7.729256026s" totalImagesPullingTime="3.458691411s" totalInitContainerRuntime="1.024637395s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:24.728898597 +0000 UTC m=+476.229803056" watchObservedRunningTime="2026-09-11 13:27:24.729256026 +0000 UTC m=+476.230160476"
Sep 11 13:27:35.710509 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:35.710476    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/opendatahub-operator-controller-manager-64959b9bbc-flljd"
Sep 11 13:27:38.118768 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.118733    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"]
Sep 11 13:27:38.127427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.127401    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.131582 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.131557    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Sep 11 13:27:38.131700 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.131672    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Sep 11 13:27:38.131745 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.131727    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-9f2q8\""
Sep 11 13:27:38.132837 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.132820    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Sep 11 13:27:38.135476 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.135459    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Sep 11 13:27:38.143754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.143730    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"]
Sep 11 13:27:38.236891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.236862    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sppls\" (UniqueName: \"kubernetes.io/projected/56cec59c-ac3b-4480-aed5-2ec531d5ed27-kube-api-access-sppls\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.237055 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.236898    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tmp\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.237055 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.236921    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tls-certs\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.337422 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.337383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sppls\" (UniqueName: \"kubernetes.io/projected/56cec59c-ac3b-4480-aed5-2ec531d5ed27-kube-api-access-sppls\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.337422 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.337423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tmp\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.337697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.337444    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tls-certs\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.339820 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.339800    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tmp\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.339951 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.339934    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/56cec59c-ac3b-4480-aed5-2ec531d5ed27-tls-certs\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.359117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.359090    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sppls\" (UniqueName: \"kubernetes.io/projected/56cec59c-ac3b-4480-aed5-2ec531d5ed27-kube-api-access-sppls\") pod \"kube-auth-proxy-8589c48f5d-9ldlx\" (UID: \"56cec59c-ac3b-4480-aed5-2ec531d5ed27\") " pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.436392 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.436295    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"
Sep 11 13:27:38.707054 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.706984    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx"]
Sep 11 13:27:38.710142 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:27:38.710115    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56cec59c_ac3b_4480_aed5_2ec531d5ed27.slice/crio-785e43f61a2d7854600f242f51e76179c929985c2fe0d3cddbc096bba9afef29 WatchSource:0}: Error finding container 785e43f61a2d7854600f242f51e76179c929985c2fe0d3cddbc096bba9afef29: Status 404 returned error can't find the container with id 785e43f61a2d7854600f242f51e76179c929985c2fe0d3cddbc096bba9afef29
Sep 11 13:27:38.752824 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:38.752789    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx" event={"ID":"56cec59c-ac3b-4480-aed5-2ec531d5ed27","Type":"ContainerStarted","Data":"785e43f61a2d7854600f242f51e76179c929985c2fe0d3cddbc096bba9afef29"}
Sep 11 13:27:39.945241 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.945204    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:27:39.948450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.948425    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:39.950855 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.950827    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Sep 11 13:27:39.951135 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.951118    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-kkk46\""
Sep 11 13:27:39.951362 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.951342    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Sep 11 13:27:39.951455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.951362    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Sep 11 13:27:39.951455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.951362    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Sep 11 13:27:39.966312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:39.966283    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:27:40.050053 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050013    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050236 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050076    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050236 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050166    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050236 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050212    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vzd5c\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050254    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050287    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.050494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.050419    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151391    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151443    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151478    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vzd5c\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151549    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151615 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151599    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.151894 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.151658    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.152266 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.152213    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.154471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.154421    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.154843 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.154820    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.154930 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.154900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.155445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.155402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.163272 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.163245    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.163805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.163781    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-vzd5c\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c\") pod \"istiod-openshift-gateway-7b7995d5cd-z7zhw\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") " pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:40.260120 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:40.260081    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:41.655529 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:41.655419    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:27:41.657939 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:27:41.657909    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a7447fd_b7af_4851_9fa4_3f897cdeda63.slice/crio-4804aedd15b9891373a67ffa736b944582ffee4e56c160f75eaff0d5170ff4be WatchSource:0}: Error finding container 4804aedd15b9891373a67ffa736b944582ffee4e56c160f75eaff0d5170ff4be: Status 404 returned error can't find the container with id 4804aedd15b9891373a67ffa736b944582ffee4e56c160f75eaff0d5170ff4be
Sep 11 13:27:41.765886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:41.765851    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx" event={"ID":"56cec59c-ac3b-4480-aed5-2ec531d5ed27","Type":"ContainerStarted","Data":"1fb27e204561ac5a1bd9f7091097d27a150fe412aa9c8cea1267426c20d5eb35"}
Sep 11 13:27:41.767063 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:41.767038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" event={"ID":"1a7447fd-b7af-4851-9fa4-3f897cdeda63","Type":"ContainerStarted","Data":"4804aedd15b9891373a67ffa736b944582ffee4e56c160f75eaff0d5170ff4be"}
Sep 11 13:27:41.787270 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:41.787227    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-8589c48f5d-9ldlx" podStartSLOduration=1.062917953 podStartE2EDuration="3.787211178s" totalImagesPullingTime="2.724293225s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:41.787047913 +0000 UTC m=+493.287952361" watchObservedRunningTime="2026-09-11 13:27:41.787211178 +0000 UTC m=+493.288115626"
Sep 11 13:27:44.242933 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:44.242888    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 13:27:44.243205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:44.243005    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 13:27:44.780719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:44.780682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" event={"ID":"1a7447fd-b7af-4851-9fa4-3f897cdeda63","Type":"ContainerStarted","Data":"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"}
Sep 11 13:27:44.780889 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:44.780741    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:44.800120 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:44.800063    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" podStartSLOduration=3.217526573 podStartE2EDuration="5.800043334s" totalImagesPullingTime="2.582516761s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:44.799402449 +0000 UTC m=+496.300306910" watchObservedRunningTime="2026-09-11 13:27:44.800043334 +0000 UTC m=+496.300947784"
Sep 11 13:27:45.786368 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:45.786333    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:49.559999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.559970    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zdqvf"]
Sep 11 13:27:49.562991 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.562973    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.568886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.568864    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-dockercfg-gkdn8\""
Sep 11 13:27:49.569349 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.569158    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"model-serving-api-tls\""
Sep 11 13:27:49.580861 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.580837    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zdqvf"]
Sep 11 13:27:49.729593 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.729555    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l8nrd\" (UniqueName: \"kubernetes.io/projected/b496961b-a099-47ec-8a34-de2690f7aefd-kube-api-access-l8nrd\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.729772 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.729654    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b496961b-a099-47ec-8a34-de2690f7aefd-tls-certs\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.830826 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.830726    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b496961b-a099-47ec-8a34-de2690f7aefd-tls-certs\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.830826 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.830779    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l8nrd\" (UniqueName: \"kubernetes.io/projected/b496961b-a099-47ec-8a34-de2690f7aefd-kube-api-access-l8nrd\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.833233 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.833209    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/b496961b-a099-47ec-8a34-de2690f7aefd-tls-certs\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.840055 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.840034    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l8nrd\" (UniqueName: \"kubernetes.io/projected/b496961b-a099-47ec-8a34-de2690f7aefd-kube-api-access-l8nrd\") pod \"model-serving-api-54d89bd5fc-zdqvf\" (UID: \"b496961b-a099-47ec-8a34-de2690f7aefd\") " pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:49.872743 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:49.872717    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:50.152008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:50.151915    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/model-serving-api-54d89bd5fc-zdqvf"]
Sep 11 13:27:50.155140 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:27:50.155110    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb496961b_a099_47ec_8a34_de2690f7aefd.slice/crio-4dd84aa59bea1fc914381a6792b58f5f5eedcd8871fd4f8174df3d2ddb740711 WatchSource:0}: Error finding container 4dd84aa59bea1fc914381a6792b58f5f5eedcd8871fd4f8174df3d2ddb740711: Status 404 returned error can't find the container with id 4dd84aa59bea1fc914381a6792b58f5f5eedcd8871fd4f8174df3d2ddb740711
Sep 11 13:27:50.805460 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:50.805420    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf" event={"ID":"b496961b-a099-47ec-8a34-de2690f7aefd","Type":"ContainerStarted","Data":"4dd84aa59bea1fc914381a6792b58f5f5eedcd8871fd4f8174df3d2ddb740711"}
Sep 11 13:27:51.810530 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:51.810485    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf" event={"ID":"b496961b-a099-47ec-8a34-de2690f7aefd","Type":"ContainerStarted","Data":"34f2ddcc4771a912b97d170aafb131578b7e8518da1b1087698c3d6d9496f664"}
Sep 11 13:27:51.810944 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:51.810587    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:27:51.833353 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:51.833304    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf" podStartSLOduration=1.391173625 podStartE2EDuration="2.833289666s" totalImagesPullingTime="1.442116041s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:51.832021613 +0000 UTC m=+503.332926075" watchObservedRunningTime="2026-09-11 13:27:51.833289666 +0000 UTC m=+503.334194114"
Sep 11 13:27:56.480588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.480549    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"]
Sep 11 13:27:56.487248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.487228    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.511036 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.511007    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"]
Sep 11 13:27:56.581140 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581106    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581140 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581142    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581198    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h5b9x\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-kube-api-access-h5b9x\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581260    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581307    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66485700-96fb-43e7-b7ce-48c63f234d47-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.581340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.581325    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682008    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682043    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682068    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h5b9x\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-kube-api-access-h5b9x\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682092    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682122    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66485700-96fb-43e7-b7ce-48c63f234d47-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682216    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.682862 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.682834    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.684452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.684419    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/66485700-96fb-43e7-b7ce-48c63f234d47-local-certs\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.684594 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.684576    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-kubeconfig\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.684594 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.684583    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-cacerts\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.684760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.684696    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/66485700-96fb-43e7-b7ce-48c63f234d47-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.694340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.694320    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-istio-token\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.694540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.694524    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-h5b9x\" (UniqueName: \"kubernetes.io/projected/66485700-96fb-43e7-b7ce-48c63f234d47-kube-api-access-h5b9x\") pod \"istiod-openshift-gateway-5677ccb49b-dbrjc\" (UID: \"66485700-96fb-43e7-b7ce-48c63f234d47\") " pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:56.795735 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:56.795654    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:57.079006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.078858    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"]
Sep 11 13:27:57.081068 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:27:57.081040    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod66485700_96fb_43e7_b7ce_48c63f234d47.slice/crio-692636194f5435a9b99162410bf5ae45653ff421a52d6e427ad08fd70e8c099f WatchSource:0}: Error finding container 692636194f5435a9b99162410bf5ae45653ff421a52d6e427ad08fd70e8c099f: Status 404 returned error can't find the container with id 692636194f5435a9b99162410bf5ae45653ff421a52d6e427ad08fd70e8c099f
Sep 11 13:27:57.084040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.084008    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 13:27:57.084116 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.084066    2551 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"31236208Ki","pods":"250"}
Sep 11 13:27:57.836293 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.836251    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc" event={"ID":"66485700-96fb-43e7-b7ce-48c63f234d47","Type":"ContainerStarted","Data":"83d5e0bdb3a282ed1e6d5994879cd07cdde4694410908e904deffe875bb47b7a"}
Sep 11 13:27:57.836862 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.836312    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc" event={"ID":"66485700-96fb-43e7-b7ce-48c63f234d47","Type":"ContainerStarted","Data":"692636194f5435a9b99162410bf5ae45653ff421a52d6e427ad08fd70e8c099f"}
Sep 11 13:27:57.836862 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.836400    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:57.878167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:57.878062    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc" podStartSLOduration=1.878042942 podStartE2EDuration="1.878042942s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:27:57.875819298 +0000 UTC m=+509.376723766" watchObservedRunningTime="2026-09-11 13:27:57.878042942 +0000 UTC m=+509.378947391"
Sep 11 13:27:58.841823 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:58.841796    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-5677ccb49b-dbrjc"
Sep 11 13:27:58.934832 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:58.934796    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:27:58.935056 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:58.935030    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" podUID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" containerName="discovery" containerID="cri-o://a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a" gracePeriod=30
Sep 11 13:27:59.176270 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.176249    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:59.197522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197490    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197534    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197558    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197647    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197679    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197906 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197707    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-vzd5c\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.197906 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197798    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert\") pod \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\" (UID: \"1a7447fd-b7af-4851-9fa4-3f897cdeda63\") "
Sep 11 13:27:59.198011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.197929    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap" (OuterVolumeSpecName: "istio-csr-ca-configmap") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:27:59.198075 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.198030    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-ca-configmap\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.200042 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200012    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c" (OuterVolumeSpecName: "kube-api-access-vzd5c") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "kube-api-access-vzd5c". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:27:59.200128 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200012    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert" (OuterVolumeSpecName: "istio-csr-dns-cert") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:27:59.200216 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200191    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs" (OuterVolumeSpecName: "local-certs") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:27:59.200286 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200232    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts" (OuterVolumeSpecName: "cacerts") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:27:59.200334 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200292    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig" (OuterVolumeSpecName: "istio-kubeconfig") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:27:59.200334 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.200299    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token" (OuterVolumeSpecName: "istio-token") pod "1a7447fd-b7af-4851-9fa4-3f897cdeda63" (UID: "1a7447fd-b7af-4851-9fa4-3f897cdeda63"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:27:59.299178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299141    2551 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-csr-dns-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.299178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299169    2551 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a7447fd-b7af-4851-9fa4-3f897cdeda63-local-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.299178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299179    2551 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-kubeconfig\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.299178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299186    2551 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-istio-token\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.299412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299194    2551 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/1a7447fd-b7af-4851-9fa4-3f897cdeda63-cacerts\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.299412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.299201    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vzd5c\" (UniqueName: \"kubernetes.io/projected/1a7447fd-b7af-4851-9fa4-3f897cdeda63-kube-api-access-vzd5c\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:27:59.845165 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.845127    2551 generic.go:356] "Generic (PLEG): container finished" podID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" containerID="a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a" exitCode=0
Sep 11 13:27:59.845165 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.845166    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"
Sep 11 13:27:59.845623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.845216    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" event={"ID":"1a7447fd-b7af-4851-9fa4-3f897cdeda63","Type":"ContainerDied","Data":"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"}
Sep 11 13:27:59.845623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.845264    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw" event={"ID":"1a7447fd-b7af-4851-9fa4-3f897cdeda63","Type":"ContainerDied","Data":"4804aedd15b9891373a67ffa736b944582ffee4e56c160f75eaff0d5170ff4be"}
Sep 11 13:27:59.845623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.845283    2551 scope.go:117] "RemoveContainer" containerID="a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"
Sep 11 13:27:59.853527 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.853512    2551 scope.go:117] "RemoveContainer" containerID="a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"
Sep 11 13:27:59.853828 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:27:59.853809    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a\": container with ID starting with a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a not found: ID does not exist" containerID="a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"
Sep 11 13:27:59.853900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.853839    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a"} err="failed to get container status \"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a\": rpc error: code = NotFound desc = could not find container \"a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a\": container with ID starting with a5dd657e5b608e6faedcab2479fa129767b44271838957ceee530ca2e495669a not found: ID does not exist"
Sep 11 13:27:59.865802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.865775    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"]
Sep 11 13:27:59.866059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.866046    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" containerName="discovery"
Sep 11 13:27:59.866124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.866060    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" containerName="discovery"
Sep 11 13:27:59.866124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.866104    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" containerName="discovery"
Sep 11 13:27:59.868581 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.868556    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:27:59.879234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.879216    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-webhook-server-cert\""
Sep 11 13:27:59.896010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.895989    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"llmisvc-controller-manager-dockercfg-hpvk7\""
Sep 11 13:27:59.901937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.901917    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6ef683-4bf0-4601-97df-f5760a05cf62-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:27:59.902612 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.901972    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zw7lp\" (UniqueName: \"kubernetes.io/projected/cd6ef683-4bf0-4601-97df-f5760a05cf62-kube-api-access-zw7lp\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:27:59.909519 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.906423    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"]
Sep 11 13:27:59.925720 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.925680    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:27:59.944619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:27:59.944591    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7b7995d5cd-z7zhw"]
Sep 11 13:28:00.003725 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.003690    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6ef683-4bf0-4601-97df-f5760a05cf62-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:00.003889 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.003743    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zw7lp\" (UniqueName: \"kubernetes.io/projected/cd6ef683-4bf0-4601-97df-f5760a05cf62-kube-api-access-zw7lp\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:00.006166 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.006131    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/cd6ef683-4bf0-4601-97df-f5760a05cf62-cert\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:00.036184 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.036148    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-zw7lp\" (UniqueName: \"kubernetes.io/projected/cd6ef683-4bf0-4601-97df-f5760a05cf62-kube-api-access-zw7lp\") pod \"llmisvc-controller-manager-777c6bd7d5-gkkmt\" (UID: \"cd6ef683-4bf0-4601-97df-f5760a05cf62\") " pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:00.177892 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.177809    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:00.570657 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.570609    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"]
Sep 11 13:28:00.573994 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:28:00.573965    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podcd6ef683_4bf0_4601_97df_f5760a05cf62.slice/crio-0928743adbb565d9771188fe46bc75dd73e2c7d8839f1d55381512bc63df2ef4 WatchSource:0}: Error finding container 0928743adbb565d9771188fe46bc75dd73e2c7d8839f1d55381512bc63df2ef4: Status 404 returned error can't find the container with id 0928743adbb565d9771188fe46bc75dd73e2c7d8839f1d55381512bc63df2ef4
Sep 11 13:28:00.850617 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.850520    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt" event={"ID":"cd6ef683-4bf0-4601-97df-f5760a05cf62","Type":"ContainerStarted","Data":"0928743adbb565d9771188fe46bc75dd73e2c7d8839f1d55381512bc63df2ef4"}
Sep 11 13:28:00.962524 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:00.962484    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1a7447fd-b7af-4851-9fa4-3f897cdeda63" path="/var/lib/kubelet/pods/1a7447fd-b7af-4851-9fa4-3f897cdeda63/volumes"
Sep 11 13:28:02.819806 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:02.819762    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/model-serving-api-54d89bd5fc-zdqvf"
Sep 11 13:28:03.866057 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:03.866020    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt" event={"ID":"cd6ef683-4bf0-4601-97df-f5760a05cf62","Type":"ContainerStarted","Data":"ecbc5fc5da77e38fec7bd81a50f1477c92c40583394cce469d1070495dc8e5d2"}
Sep 11 13:28:03.866473 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:03.866114    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:28:03.888086 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:03.888027    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt" podStartSLOduration=2.489100833 podStartE2EDuration="4.888006038s" totalImagesPullingTime="2.398905205s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:27:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:28:03.886596901 +0000 UTC m=+515.387501350" watchObservedRunningTime="2026-09-11 13:28:03.888006038 +0000 UTC m=+515.388910488"
Sep 11 13:28:34.872332 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:28:34.872302    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/llmisvc-controller-manager-777c6bd7d5-gkkmt"
Sep 11 13:29:11.322806 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.322767    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"]
Sep 11 13:29:11.325547 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.325532    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:11.329070 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.329051    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Sep 11 13:29:11.333792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.333775    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Sep 11 13:29:11.343899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.343883    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"limitador-operator-controller-manager-dockercfg-4fsn7\""
Sep 11 13:29:11.356969 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.356940    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"]
Sep 11 13:29:11.358780 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.358760    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7kzk5\" (UniqueName: \"kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5\") pod \"limitador-operator-controller-manager-74495c69f8-2l4pf\" (UID: \"e444e6c0-7a13-4d55-9a4d-f60adc20f933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:11.459865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.459827    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7kzk5\" (UniqueName: \"kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5\") pod \"limitador-operator-controller-manager-74495c69f8-2l4pf\" (UID: \"e444e6c0-7a13-4d55-9a4d-f60adc20f933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:11.500313 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.500268    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7kzk5\" (UniqueName: \"kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5\") pod \"limitador-operator-controller-manager-74495c69f8-2l4pf\" (UID: \"e444e6c0-7a13-4d55-9a4d-f60adc20f933\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:11.635549 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.635469    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:11.929796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:11.929718    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"]
Sep 11 13:29:11.933243 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:29:11.933213    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode444e6c0_7a13_4d55_9a4d_f60adc20f933.slice/crio-287d65a11a58d75329749bd893c1ab3f3872c12268944ff093ba6c0f125000ff WatchSource:0}: Error finding container 287d65a11a58d75329749bd893c1ab3f3872c12268944ff093ba6c0f125000ff: Status 404 returned error can't find the container with id 287d65a11a58d75329749bd893c1ab3f3872c12268944ff093ba6c0f125000ff
Sep 11 13:29:12.122512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:12.122479    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf" event={"ID":"e444e6c0-7a13-4d55-9a4d-f60adc20f933","Type":"ContainerStarted","Data":"287d65a11a58d75329749bd893c1ab3f3872c12268944ff093ba6c0f125000ff"}
Sep 11 13:29:16.145046 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:16.145011    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf" event={"ID":"e444e6c0-7a13-4d55-9a4d-f60adc20f933","Type":"ContainerStarted","Data":"cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006"}
Sep 11 13:29:16.145456 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:16.145157    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:16.167199 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:16.167150    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf" podStartSLOduration=1.800726343 podStartE2EDuration="5.167135415s" totalImagesPullingTime="3.366409072s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:11 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:16.165408732 +0000 UTC m=+587.666313203" watchObservedRunningTime="2026-09-11 13:29:16.167135415 +0000 UTC m=+587.668039863"
Sep 11 13:29:17.770491 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.770451    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"]
Sep 11 13:29:17.773693 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.773673    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:17.777245 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.777223    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-env\""
Sep 11 13:29:17.777338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.777292    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"dns-operator-controller-manager-dockercfg-f5kx9\""
Sep 11 13:29:17.792412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.792379    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"]
Sep 11 13:29:17.812248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.812215    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/dc5ed2da-05ed-4b48-b322-e84b69bfe6e1-kube-api-access-tcwkl\") pod \"dns-operator-controller-manager-7b84cd7c9c-jw59x\" (UID: \"dc5ed2da-05ed-4b48-b322-e84b69bfe6e1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:17.913075 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.913040    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/dc5ed2da-05ed-4b48-b322-e84b69bfe6e1-kube-api-access-tcwkl\") pod \"dns-operator-controller-manager-7b84cd7c9c-jw59x\" (UID: \"dc5ed2da-05ed-4b48-b322-e84b69bfe6e1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:17.927455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:17.927424    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tcwkl\" (UniqueName: \"kubernetes.io/projected/dc5ed2da-05ed-4b48-b322-e84b69bfe6e1-kube-api-access-tcwkl\") pod \"dns-operator-controller-manager-7b84cd7c9c-jw59x\" (UID: \"dc5ed2da-05ed-4b48-b322-e84b69bfe6e1\") " pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:18.083757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:18.083660    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:18.364352 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:18.364327    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"]
Sep 11 13:29:18.368053 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:29:18.368027    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc5ed2da_05ed_4b48_b322_e84b69bfe6e1.slice/crio-fcd25a7b42ca48c0b370fafb4161619ca80ff4407d8ca64730f4593925287149 WatchSource:0}: Error finding container fcd25a7b42ca48c0b370fafb4161619ca80ff4407d8ca64730f4593925287149: Status 404 returned error can't find the container with id fcd25a7b42ca48c0b370fafb4161619ca80ff4407d8ca64730f4593925287149
Sep 11 13:29:19.160039 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:19.160001    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x" event={"ID":"dc5ed2da-05ed-4b48-b322-e84b69bfe6e1","Type":"ContainerStarted","Data":"fcd25a7b42ca48c0b370fafb4161619ca80ff4407d8ca64730f4593925287149"}
Sep 11 13:29:20.165256 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:20.165218    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x" event={"ID":"dc5ed2da-05ed-4b48-b322-e84b69bfe6e1","Type":"ContainerStarted","Data":"3bc78978dda8ce3fcc1c7dcba67d74f4be25fa45812c97838226fb672483f54b"}
Sep 11 13:29:20.165621 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:20.165369    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:20.199599 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:20.199546    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x" podStartSLOduration=2.003598833 podStartE2EDuration="3.199528984s" totalImagesPullingTime="1.195930151s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:17 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:20.198975592 +0000 UTC m=+591.699880066" watchObservedRunningTime="2026-09-11 13:29:20.199528984 +0000 UTC m=+591.700433433"
Sep 11 13:29:27.151180 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:27.151145    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:31.171388 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:31.171358    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/dns-operator-controller-manager-7b84cd7c9c-jw59x"
Sep 11 13:29:34.411334 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.411298    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"]
Sep 11 13:29:34.411745 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.411516    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" containerName="manager" containerID="cri-o://cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006" gracePeriod=2
Sep 11 13:29:34.445983 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.445955    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"]
Sep 11 13:29:34.574557 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.574518    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"]
Sep 11 13:29:34.574968 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.574952    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" containerName="manager"
Sep 11 13:29:34.575018 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.574971    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" containerName="manager"
Sep 11 13:29:34.575059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.575043    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" containerName="manager"
Sep 11 13:29:34.578355 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.578337    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:34.585500 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:29:34.585469    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-2l4pf\" is forbidden: User \"system:node:ip-10-0-141-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-50.ec2.internal' and this object" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:34.595181 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.595098    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"]
Sep 11 13:29:34.639019 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.638991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7q7s9\" (UniqueName: \"kubernetes.io/projected/56080ba9-e22a-48f7-89ab-6318e6f843e1-kube-api-access-7q7s9\") pod \"limitador-operator-controller-manager-74495c69f8-rpfwg\" (UID: \"56080ba9-e22a-48f7-89ab-6318e6f843e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:34.652995 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.652971    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:34.655354 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:29:34.655331    2551 status_manager.go:1018] "Failed to get status for pod" err="pods \"limitador-operator-controller-manager-74495c69f8-2l4pf\" is forbidden: User \"system:node:ip-10-0-141-50.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"kuadrant-system\": no relationship found between node 'ip-10-0-141-50.ec2.internal' and this object" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:34.739671 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.739618    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-7kzk5\" (UniqueName: \"kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5\") pod \"e444e6c0-7a13-4d55-9a4d-f60adc20f933\" (UID: \"e444e6c0-7a13-4d55-9a4d-f60adc20f933\") "
Sep 11 13:29:34.739832 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.739816    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7q7s9\" (UniqueName: \"kubernetes.io/projected/56080ba9-e22a-48f7-89ab-6318e6f843e1-kube-api-access-7q7s9\") pod \"limitador-operator-controller-manager-74495c69f8-rpfwg\" (UID: \"56080ba9-e22a-48f7-89ab-6318e6f843e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:34.741845 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.741819    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5" (OuterVolumeSpecName: "kube-api-access-7kzk5") pod "e444e6c0-7a13-4d55-9a4d-f60adc20f933" (UID: "e444e6c0-7a13-4d55-9a4d-f60adc20f933"). InnerVolumeSpecName "kube-api-access-7kzk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:29:34.756702 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.756680    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-7q7s9\" (UniqueName: \"kubernetes.io/projected/56080ba9-e22a-48f7-89ab-6318e6f843e1-kube-api-access-7q7s9\") pod \"limitador-operator-controller-manager-74495c69f8-rpfwg\" (UID: \"56080ba9-e22a-48f7-89ab-6318e6f843e1\") " pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:34.840527 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.840487    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7kzk5\" (UniqueName: \"kubernetes.io/projected/e444e6c0-7a13-4d55-9a4d-f60adc20f933-kube-api-access-7kzk5\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:29:34.889308 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.889281    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:34.964973 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:34.964937    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" path="/var/lib/kubelet/pods/e444e6c0-7a13-4d55-9a4d-f60adc20f933/volumes"
Sep 11 13:29:35.219275 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.219252    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"]
Sep 11 13:29:35.221297 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:29:35.221273    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56080ba9_e22a_48f7_89ab_6318e6f843e1.slice/crio-70993bbcf23366e46c2e1b9dc157d4a38241195b4d433ee7150617175bb23f59 WatchSource:0}: Error finding container 70993bbcf23366e46c2e1b9dc157d4a38241195b4d433ee7150617175bb23f59: Status 404 returned error can't find the container with id 70993bbcf23366e46c2e1b9dc157d4a38241195b4d433ee7150617175bb23f59
Sep 11 13:29:35.227814 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.227789    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg" event={"ID":"56080ba9-e22a-48f7-89ab-6318e6f843e1","Type":"ContainerStarted","Data":"70993bbcf23366e46c2e1b9dc157d4a38241195b4d433ee7150617175bb23f59"}
Sep 11 13:29:35.229222 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.229196    2551 generic.go:356] "Generic (PLEG): container finished" podID="e444e6c0-7a13-4d55-9a4d-f60adc20f933" containerID="cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006" exitCode=0
Sep 11 13:29:35.229304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.229232    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-2l4pf"
Sep 11 13:29:35.229304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.229289    2551 scope.go:117] "RemoveContainer" containerID="cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006"
Sep 11 13:29:35.236745 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.236729    2551 scope.go:117] "RemoveContainer" containerID="cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006"
Sep 11 13:29:35.237087 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:29:35.237060    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006\": container with ID starting with cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006 not found: ID does not exist" containerID="cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006"
Sep 11 13:29:35.237156 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:35.237098    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006"} err="failed to get container status \"cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006\": rpc error: code = NotFound desc = could not find container \"cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006\": container with ID starting with cc2750e3841c3af8d42ab23e1fa67099b1aba1054ce9a1bb443cc2d86d047006 not found: ID does not exist"
Sep 11 13:29:36.236451 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:36.236413    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg" event={"ID":"56080ba9-e22a-48f7-89ab-6318e6f843e1","Type":"ContainerStarted","Data":"49dc822dd8606e1e77f9787da3f255551c3a9ba58a340659f76d0be6b14174e1"}
Sep 11 13:29:36.236918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:36.236516    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:37.763235 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.763175    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg" podStartSLOduration=3.763157282 podStartE2EDuration="3.763157282s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:36.267664795 +0000 UTC m=+607.768569258" watchObservedRunningTime="2026-09-11 13:29:37.763157282 +0000 UTC m=+609.264061730"
Sep 11 13:29:37.763753 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.763463    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:29:37.766548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.766527    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.773438 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.773418    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m7xkw\""
Sep 11 13:29:37.798441 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.798413    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:29:37.866431 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.866394    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sc7zd\" (UniqueName: \"kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.866610 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.866489    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.967281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.967240    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.967457 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.967313    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sc7zd\" (UniqueName: \"kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.967614 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.967590    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:37.983520 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:37.983497    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sc7zd\" (UniqueName: \"kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd\") pod \"kuadrant-operator-controller-manager-5769f99d9f-9lc58\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:38.076427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:38.076346    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:38.354971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:38.354942    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:29:38.356499 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:29:38.356477    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb50668f2_df96_4ef8_82f9_5729217169ae.slice/crio-672fdd666492e7e2fe3c5b7723c6c47f1c1383f297096b0077d8603c7475f299 WatchSource:0}: Error finding container 672fdd666492e7e2fe3c5b7723c6c47f1c1383f297096b0077d8603c7475f299: Status 404 returned error can't find the container with id 672fdd666492e7e2fe3c5b7723c6c47f1c1383f297096b0077d8603c7475f299
Sep 11 13:29:39.253174 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:39.253131    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" event={"ID":"b50668f2-df96-4ef8-82f9-5729217169ae","Type":"ContainerStarted","Data":"672fdd666492e7e2fe3c5b7723c6c47f1c1383f297096b0077d8603c7475f299"}
Sep 11 13:29:42.271563 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:42.271479    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" event={"ID":"b50668f2-df96-4ef8-82f9-5729217169ae","Type":"ContainerStarted","Data":"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"}
Sep 11 13:29:42.272118 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:42.271603    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:29:42.372954 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:42.372895    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" podStartSLOduration=1.850861073 podStartE2EDuration="5.372878295s" totalImagesPullingTime="3.522017222s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:29:42.348399688 +0000 UTC m=+613.849304138" watchObservedRunningTime="2026-09-11 13:29:42.372878295 +0000 UTC m=+613.873782743"
Sep 11 13:29:47.244710 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:47.244679    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-operator-controller-manager-74495c69f8-rpfwg"
Sep 11 13:29:53.277748 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:29:53.277675    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:30:13.527001 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.526959    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:13.530433 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.530409    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:13.532762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.532735    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-l8kvx\""
Sep 11 13:30:13.539061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.539037    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:13.548100 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.548073    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l564j\" (UniqueName: \"kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j\") pod \"authorino-7bcfd8f9cd-84kwd\" (UID: \"3a9e0c38-0cf2-428e-83d9-1243e81800bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:13.649044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.649008    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l564j\" (UniqueName: \"kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j\") pod \"authorino-7bcfd8f9cd-84kwd\" (UID: \"3a9e0c38-0cf2-428e-83d9-1243e81800bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:13.660708 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.660682    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-l564j\" (UniqueName: \"kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j\") pod \"authorino-7bcfd8f9cd-84kwd\" (UID: \"3a9e0c38-0cf2-428e-83d9-1243e81800bf\") " pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:13.840818 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:13.840738    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:14.119441 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:14.119367    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:14.122565 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:30:14.122527    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a9e0c38_0cf2_428e_83d9_1243e81800bf.slice/crio-da59314bba77024c85e11c65c30ddc0818b90b2540bcef1d6cb41f4aec262bea WatchSource:0}: Error finding container da59314bba77024c85e11c65c30ddc0818b90b2540bcef1d6cb41f4aec262bea: Status 404 returned error can't find the container with id da59314bba77024c85e11c65c30ddc0818b90b2540bcef1d6cb41f4aec262bea
Sep 11 13:30:14.407040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:14.406955    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" event={"ID":"3a9e0c38-0cf2-428e-83d9-1243e81800bf","Type":"ContainerStarted","Data":"da59314bba77024c85e11c65c30ddc0818b90b2540bcef1d6cb41f4aec262bea"}
Sep 11 13:30:16.418763 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:16.418677    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" event={"ID":"3a9e0c38-0cf2-428e-83d9-1243e81800bf","Type":"ContainerStarted","Data":"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"}
Sep 11 13:30:16.439059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:16.439010    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" podStartSLOduration=1.423847849 podStartE2EDuration="3.438992739s" totalImagesPullingTime="2.01514489s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:30:16.437081862 +0000 UTC m=+647.937986311" watchObservedRunningTime="2026-09-11 13:30:16.438992739 +0000 UTC m=+647.939897217"
Sep 11 13:30:17.430094 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:17.430062    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:18.426606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:18.426563    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" podUID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" containerName="authorino" containerID="cri-o://b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f" gracePeriod=30
Sep 11 13:30:18.669248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:18.669221    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:18.679406 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:18.679347    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-l564j\" (UniqueName: \"kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j\") pod \"3a9e0c38-0cf2-428e-83d9-1243e81800bf\" (UID: \"3a9e0c38-0cf2-428e-83d9-1243e81800bf\") "
Sep 11 13:30:18.681204 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:18.681179    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j" (OuterVolumeSpecName: "kube-api-access-l564j") pod "3a9e0c38-0cf2-428e-83d9-1243e81800bf" (UID: "3a9e0c38-0cf2-428e-83d9-1243e81800bf"). InnerVolumeSpecName "kube-api-access-l564j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:30:18.780435 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:18.780396    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l564j\" (UniqueName: \"kubernetes.io/projected/3a9e0c38-0cf2-428e-83d9-1243e81800bf-kube-api-access-l564j\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:30:19.431926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.431837    2551 generic.go:356] "Generic (PLEG): container finished" podID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" containerID="b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f" exitCode=0
Sep 11 13:30:19.431926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.431898    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" event={"ID":"3a9e0c38-0cf2-428e-83d9-1243e81800bf","Type":"ContainerDied","Data":"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"}
Sep 11 13:30:19.431926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.431925    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd" event={"ID":"3a9e0c38-0cf2-428e-83d9-1243e81800bf","Type":"ContainerDied","Data":"da59314bba77024c85e11c65c30ddc0818b90b2540bcef1d6cb41f4aec262bea"}
Sep 11 13:30:19.432148 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.431942    2551 scope.go:117] "RemoveContainer" containerID="b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"
Sep 11 13:30:19.432148 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.431900    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-7bcfd8f9cd-84kwd"
Sep 11 13:30:19.439939 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.439916    2551 scope.go:117] "RemoveContainer" containerID="b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"
Sep 11 13:30:19.440199 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:30:19.440181    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f\": container with ID starting with b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f not found: ID does not exist" containerID="b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"
Sep 11 13:30:19.440248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.440208    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f"} err="failed to get container status \"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f\": rpc error: code = NotFound desc = could not find container \"b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f\": container with ID starting with b5ac7b314e3cb00d53ce0aa01c8cb106149964f2b0ecad4f546f19de3a8ad88f not found: ID does not exist"
Sep 11 13:30:19.448010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.447986    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:19.451877 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:19.451857    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-7bcfd8f9cd-84kwd"]
Sep 11 13:30:20.963204 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:20.963158    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" path="/var/lib/kubelet/pods/3a9e0c38-0cf2-428e-83d9-1243e81800bf/volumes"
Sep 11 13:30:56.684463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.684422    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:30:56.684951 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.684715    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" containerName="authorino"
Sep 11 13:30:56.684951 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.684727    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" containerName="authorino"
Sep 11 13:30:56.684951 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.684781    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="3a9e0c38-0cf2-428e-83d9-1243e81800bf" containerName="authorino"
Sep 11 13:30:56.687565 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.687543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.690295 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.690274    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 13:30:56.691416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.691383    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 13:30:56.691416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.691404    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4q2dl\""
Sep 11 13:30:56.691596 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.691472    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 13:30:56.699189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.699166    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:30:56.775729 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.775693    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.775913 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.775747    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gqkmq\" (UniqueName: \"kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.775913 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.775770    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.876821 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.876783    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gqkmq\" (UniqueName: \"kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.876971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.876939    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.877035 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.877021    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.877144 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:30:56.877132    2551 secret.go:189] Couldn't get secret opendatahub/maas-controller-webhook-cert: secret "maas-controller-webhook-cert" not found
Sep 11 13:30:56.877200 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:30:56.877191    2551 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert podName:8025e512-66b9-4d2e-99e1-1fe05dcc65d7 nodeName:}" failed. No retries permitted until 2026-09-11 13:30:57.377174868 +0000 UTC m=+688.878079297 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "webhook-cert" (UniqueName: "kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert") pod "maas-controller-59798b69f7-z4p2z" (UID: "8025e512-66b9-4d2e-99e1-1fe05dcc65d7") : secret "maas-controller-webhook-cert" not found
Sep 11 13:30:56.879526 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.879507    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:56.889129 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:56.889107    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gqkmq\" (UniqueName: \"kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:57.381146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:57.381108    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:57.383576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:57.383552    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") pod \"maas-controller-59798b69f7-z4p2z\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") " pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:57.597526 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:57.597491    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:30:57.869047 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:57.869012    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:30:57.872336 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:30:57.872301    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8025e512_66b9_4d2e_99e1_1fe05dcc65d7.slice/crio-6c1fd0cbf0b05fe0dc01184c858ac5c1a4b1e30cc207a2736df0439503f21686 WatchSource:0}: Error finding container 6c1fd0cbf0b05fe0dc01184c858ac5c1a4b1e30cc207a2736df0439503f21686: Status 404 returned error can't find the container with id 6c1fd0cbf0b05fe0dc01184c858ac5c1a4b1e30cc207a2736df0439503f21686
Sep 11 13:30:58.598331 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:30:58.598296    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59798b69f7-z4p2z" event={"ID":"8025e512-66b9-4d2e-99e1-1fe05dcc65d7","Type":"ContainerStarted","Data":"6c1fd0cbf0b05fe0dc01184c858ac5c1a4b1e30cc207a2736df0439503f21686"}
Sep 11 13:31:00.609701 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:00.609660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59798b69f7-z4p2z" event={"ID":"8025e512-66b9-4d2e-99e1-1fe05dcc65d7","Type":"ContainerStarted","Data":"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"}
Sep 11 13:31:00.610164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:00.609822    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:31:00.647395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:00.647329    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-59798b69f7-z4p2z" podStartSLOduration=1.99741361 podStartE2EDuration="4.647313744s" totalImagesPullingTime="2.649900134s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:30:56 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:00.646193046 +0000 UTC m=+692.147097513" watchObservedRunningTime="2026-09-11 13:31:00.647313744 +0000 UTC m=+692.148218192"
Sep 11 13:31:11.617041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:11.617010    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:31:11.853098 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:11.853060    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:31:11.853335 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:11.853312    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-59798b69f7-z4p2z" podUID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" containerName="manager" containerID="cri-o://a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558" gracePeriod=30
Sep 11 13:31:12.088653 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.088614    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:31:12.202148 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.202060    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") pod \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") "
Sep 11 13:31:12.202305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.202161    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gqkmq\" (UniqueName: \"kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq\") pod \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") "
Sep 11 13:31:12.202305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.202192    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs\") pod \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\" (UID: \"8025e512-66b9-4d2e-99e1-1fe05dcc65d7\") "
Sep 11 13:31:12.204315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.204281    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "8025e512-66b9-4d2e-99e1-1fe05dcc65d7" (UID: "8025e512-66b9-4d2e-99e1-1fe05dcc65d7"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:31:12.204419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.204363    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq" (OuterVolumeSpecName: "kube-api-access-gqkmq") pod "8025e512-66b9-4d2e-99e1-1fe05dcc65d7" (UID: "8025e512-66b9-4d2e-99e1-1fe05dcc65d7"). InnerVolumeSpecName "kube-api-access-gqkmq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:31:12.204419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.204383    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "8025e512-66b9-4d2e-99e1-1fe05dcc65d7" (UID: "8025e512-66b9-4d2e-99e1-1fe05dcc65d7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:31:12.302971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.302929    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gqkmq\" (UniqueName: \"kubernetes.io/projected/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-kube-api-access-gqkmq\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:31:12.302971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.302971    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-metrics-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:31:12.303169 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.302987    2551 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/8025e512-66b9-4d2e-99e1-1fe05dcc65d7-webhook-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:31:12.661037 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.660998    2551 generic.go:356] "Generic (PLEG): container finished" podID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" containerID="a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558" exitCode=0
Sep 11 13:31:12.661436 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.661048    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-59798b69f7-z4p2z"
Sep 11 13:31:12.661436 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.661054    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59798b69f7-z4p2z" event={"ID":"8025e512-66b9-4d2e-99e1-1fe05dcc65d7","Type":"ContainerDied","Data":"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"}
Sep 11 13:31:12.661436 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.661106    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-59798b69f7-z4p2z" event={"ID":"8025e512-66b9-4d2e-99e1-1fe05dcc65d7","Type":"ContainerDied","Data":"6c1fd0cbf0b05fe0dc01184c858ac5c1a4b1e30cc207a2736df0439503f21686"}
Sep 11 13:31:12.661436 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.661136    2551 scope.go:117] "RemoveContainer" containerID="a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"
Sep 11 13:31:12.669574 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.669555    2551 scope.go:117] "RemoveContainer" containerID="a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"
Sep 11 13:31:12.669884 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:31:12.669864    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558\": container with ID starting with a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558 not found: ID does not exist" containerID="a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"
Sep 11 13:31:12.669953 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.669892    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558"} err="failed to get container status \"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558\": rpc error: code = NotFound desc = could not find container \"a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558\": container with ID starting with a24d997712e46f9e34e379ab84f03db847db99b645ad2daaeceff6c36096e558 not found: ID does not exist"
Sep 11 13:31:12.682801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.682779    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:31:12.687660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.687613    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-59798b69f7-z4p2z"]
Sep 11 13:31:12.963439 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:12.963363    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" path="/var/lib/kubelet/pods/8025e512-66b9-4d2e-99e1-1fe05dcc65d7/volumes"
Sep 11 13:31:34.662844 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.662809    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"]
Sep 11 13:31:34.663232 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.663085    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" containerName="manager"
Sep 11 13:31:34.663232 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.663096    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" containerName="manager"
Sep 11 13:31:34.663232 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.663152    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8025e512-66b9-4d2e-99e1-1fe05dcc65d7" containerName="manager"
Sep 11 13:31:34.666738 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.666717    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.669553 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.669519    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins\""
Sep 11 13:31:34.714404 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.714371    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"]
Sep 11 13:31:34.776595 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.776557    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.776812 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.776741    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkvk8\" (UniqueName: \"kubernetes.io/projected/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-kube-api-access-hkvk8\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.778651 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.778613    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-k65dc"]
Sep 11 13:31:34.780781 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.780766    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.783940 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.783921    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-dockercfg-tbr8l\""
Sep 11 13:31:34.809338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.809300    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-k65dc"]
Sep 11 13:31:34.877282 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.877252    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.877431 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.877311    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dwfdg\" (UniqueName: \"kubernetes.io/projected/c2ebb4a2-0943-4a41-bfbc-449854d04c77-kube-api-access-dwfdg\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.877431 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.877340    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebb4a2-0943-4a41-bfbc-449854d04c77-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.877431 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.877374    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkvk8\" (UniqueName: \"kubernetes.io/projected/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-kube-api-access-hkvk8\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.877965 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.877944    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-plugins-config-volume\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.893757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.893731    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkvk8\" (UniqueName: \"kubernetes.io/projected/c2f407a1-0426-4f25-bea0-9aa10bbc6e14-kube-api-access-hkvk8\") pod \"payload-pre-processing-59fbc47bf8-k7gg4\" (UID: \"c2f407a1-0426-4f25-bea0-9aa10bbc6e14\") " pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.977157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.977129    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:34.978453 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.978426    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dwfdg\" (UniqueName: \"kubernetes.io/projected/c2ebb4a2-0943-4a41-bfbc-449854d04c77-kube-api-access-dwfdg\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.978597 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.978470    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebb4a2-0943-4a41-bfbc-449854d04c77-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.979449 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.979155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2ebb4a2-0943-4a41-bfbc-449854d04c77-plugins-config-volume\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:34.993358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:34.993330    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dwfdg\" (UniqueName: \"kubernetes.io/projected/c2ebb4a2-0943-4a41-bfbc-449854d04c77-kube-api-access-dwfdg\") pod \"payload-processing-5b54b7fbb7-k65dc\" (UID: \"c2ebb4a2-0943-4a41-bfbc-449854d04c77\") " pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:35.090274 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:35.090240    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:35.262678 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:35.262485    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"]
Sep 11 13:31:35.266245 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:31:35.266211    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2f407a1_0426_4f25_bea0_9aa10bbc6e14.slice/crio-1d5d82863a9ec4fc67565612b91aa7d409971e24181b1de5014bac3ad35b9d26 WatchSource:0}: Error finding container 1d5d82863a9ec4fc67565612b91aa7d409971e24181b1de5014bac3ad35b9d26: Status 404 returned error can't find the container with id 1d5d82863a9ec4fc67565612b91aa7d409971e24181b1de5014bac3ad35b9d26
Sep 11 13:31:35.369548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:35.369523    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-5b54b7fbb7-k65dc"]
Sep 11 13:31:35.371920 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:31:35.371897    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2ebb4a2_0943_4a41_bfbc_449854d04c77.slice/crio-aa87a60cb70bfcf51d86ef3d9af716543de75c1ce09cc0f174fe0b39a5ab77fa WatchSource:0}: Error finding container aa87a60cb70bfcf51d86ef3d9af716543de75c1ce09cc0f174fe0b39a5ab77fa: Status 404 returned error can't find the container with id aa87a60cb70bfcf51d86ef3d9af716543de75c1ce09cc0f174fe0b39a5ab77fa
Sep 11 13:31:35.753398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:35.753365    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4" event={"ID":"c2f407a1-0426-4f25-bea0-9aa10bbc6e14","Type":"ContainerStarted","Data":"1d5d82863a9ec4fc67565612b91aa7d409971e24181b1de5014bac3ad35b9d26"}
Sep 11 13:31:35.754589 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:35.754565    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc" event={"ID":"c2ebb4a2-0943-4a41-bfbc-449854d04c77","Type":"ContainerStarted","Data":"aa87a60cb70bfcf51d86ef3d9af716543de75c1ce09cc0f174fe0b39a5ab77fa"}
Sep 11 13:31:37.773291 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.773250    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc" event={"ID":"c2ebb4a2-0943-4a41-bfbc-449854d04c77","Type":"ContainerStarted","Data":"94a37ad8743ebd08996bba12dbdb0ea550908dc6c97519649cfd73fb17f95326"}
Sep 11 13:31:37.773783 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.773363    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:31:37.774997 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.774973    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4" event={"ID":"c2f407a1-0426-4f25-bea0-9aa10bbc6e14","Type":"ContainerStarted","Data":"7b3145541517225c590488cedd2f3cb70557e220f0620e10c254661b97f7aa09"}
Sep 11 13:31:37.775116 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.775104    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:37.796880 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.796829    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc" podStartSLOduration=2.12427991 podStartE2EDuration="3.796814812s" totalImagesPullingTime="1.672534902s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:37.794163074 +0000 UTC m=+729.295067534" watchObservedRunningTime="2026-09-11 13:31:37.796814812 +0000 UTC m=+729.297719260"
Sep 11 13:31:37.813773 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:37.813729    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4" podStartSLOduration=2.026248482 podStartE2EDuration="3.813713213s" totalImagesPullingTime="1.787464731s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:31:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:31:37.812822702 +0000 UTC m=+729.313727167" watchObservedRunningTime="2026-09-11 13:31:37.813713213 +0000 UTC m=+729.314617662"
Sep 11 13:31:48.782216 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:48.782187    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-59fbc47bf8-k7gg4"
Sep 11 13:31:48.782583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:31:48.782255    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-5b54b7fbb7-k65dc"
Sep 11 13:32:04.813025 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.812993    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"]
Sep 11 13:32:04.815339 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.815318    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.817898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.817877    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Sep 11 13:32:04.818662 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.818638    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-simulated-kserve-self-signed-certs\""
Sep 11 13:32:04.818662 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.818656    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-zv5p8\""
Sep 11 13:32:04.818814 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.818659    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Sep 11 13:32:04.827770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.827747    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"]
Sep 11 13:32:04.929980 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.929933    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.930185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.929997    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.930185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.930065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec359d7-9c83-4116-9654-0a7541d9f843-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.930185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.930113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.930185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.930163    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v2kdr\" (UniqueName: \"kubernetes.io/projected/eec359d7-9c83-4116-9654-0a7541d9f843-kube-api-access-v2kdr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:04.930447 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:04.930214    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031379    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031422    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v2kdr\" (UniqueName: \"kubernetes.io/projected/eec359d7-9c83-4116-9654-0a7541d9f843-kube-api-access-v2kdr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031450    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031477    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031496    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031719 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031530    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec359d7-9c83-4116-9654-0a7541d9f843-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031963 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031888    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-model-cache\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.031963 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031908    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-home\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.032027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.031963    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-tmp-dir\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.033746 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.033720    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/eec359d7-9c83-4116-9654-0a7541d9f843-dshm\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.034013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.033988    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/eec359d7-9c83-4116-9654-0a7541d9f843-tls-certs\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.040102 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.040080    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-v2kdr\" (UniqueName: \"kubernetes.io/projected/eec359d7-9c83-4116-9654-0a7541d9f843-kube-api-access-v2kdr\") pod \"e2e-distinct-simulated-kserve-59975d69f9-bfpdr\" (UID: \"eec359d7-9c83-4116-9654-0a7541d9f843\") " pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.125818 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.125732    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:05.402422 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.402370    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"]
Sep 11 13:32:05.405304 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:32:05.405275    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podeec359d7_9c83_4116_9654_0a7541d9f843.slice/crio-25d847e30bf96555e196c7cd31e747f7714c21cfb3e066c94cff7bb497852613 WatchSource:0}: Error finding container 25d847e30bf96555e196c7cd31e747f7714c21cfb3e066c94cff7bb497852613: Status 404 returned error can't find the container with id 25d847e30bf96555e196c7cd31e747f7714c21cfb3e066c94cff7bb497852613
Sep 11 13:32:05.406755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.406736    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:32:05.896223 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:05.896180    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr" event={"ID":"eec359d7-9c83-4116-9654-0a7541d9f843","Type":"ContainerStarted","Data":"25d847e30bf96555e196c7cd31e747f7714c21cfb3e066c94cff7bb497852613"}
Sep 11 13:32:06.902256 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:06.902225    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr" event={"ID":"eec359d7-9c83-4116-9654-0a7541d9f843","Type":"ContainerStarted","Data":"ee498f453c1674e6efd3cd66f457acbf55cee30c0080fcab9fff3dd1981086ca"}
Sep 11 13:32:06.921289 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:06.921238    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr" podStartSLOduration=1.940525467 podStartE2EDuration="2.921221638s" totalImagesPullingTime="980.696171ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:32:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:06.919617984 +0000 UTC m=+758.420522445" watchObservedRunningTime="2026-09-11 13:32:06.921221638 +0000 UTC m=+758.422126130"
Sep 11 13:32:15.126740 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:15.126701    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:15.127140 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:15.126767    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:15.139264 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:15.139241    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:15.951580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:15.951551    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-simulated-kserve-59975d69f9-bfpdr"
Sep 11 13:32:42.028315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.028284    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"]
Sep 11 13:32:42.030746 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.030718    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.034080 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.034060    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-unab60ef4d3a239b5143b412cab04acac3-kserve-self-signed-certs\""
Sep 11 13:32:42.047462 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.047430    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"]
Sep 11 13:32:42.125583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125545    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.125583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125585    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.125810 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125614    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77d024c1-df3b-47c9-a905-c64ec06abfac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.125810 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125702    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.125810 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125740    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4v2nt\" (UniqueName: \"kubernetes.io/projected/77d024c1-df3b-47c9-a905-c64ec06abfac-kube-api-access-4v2nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.125810 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.125778    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226268 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226228    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77d024c1-df3b-47c9-a905-c64ec06abfac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226466 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226284    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226466 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226325    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4v2nt\" (UniqueName: \"kubernetes.io/projected/77d024c1-df3b-47c9-a905-c64ec06abfac-kube-api-access-4v2nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226466 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226373    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226466 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226460    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226840 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226486    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226840 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226811    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-model-cache\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.226957 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.226856    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-home\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.227045 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.227021    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-tmp-dir\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.228915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.228890    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/77d024c1-df3b-47c9-a905-c64ec06abfac-dshm\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.228915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.228900    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/77d024c1-df3b-47c9-a905-c64ec06abfac-tls-certs\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.236117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.236095    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4v2nt\" (UniqueName: \"kubernetes.io/projected/77d024c1-df3b-47c9-a905-c64ec06abfac-kube-api-access-4v2nt\") pod \"e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn\" (UID: \"77d024c1-df3b-47c9-a905-c64ec06abfac\") " pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.340365 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.340278    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:42.619761 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:42.616587    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"]
Sep 11 13:32:43.062684 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:43.062623    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn" event={"ID":"77d024c1-df3b-47c9-a905-c64ec06abfac","Type":"ContainerStarted","Data":"e062c28123b48e4c35a0d22738fe10f325654a26a0dad41884378cc5e07540b1"}
Sep 11 13:32:43.063094 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:43.062705    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn" event={"ID":"77d024c1-df3b-47c9-a905-c64ec06abfac","Type":"ContainerStarted","Data":"9d3f40048daaf43172f07e08c1a45f54bec8e5226d106124ad645a086db7d5bf"}
Sep 11 13:32:43.087032 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:43.086965    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn" podStartSLOduration=1.903602907 podStartE2EDuration="2.086943496s" totalImagesPullingTime="183.340589ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:32:41 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:32:43.084735328 +0000 UTC m=+794.585639792" watchObservedRunningTime="2026-09-11 13:32:43.086943496 +0000 UTC m=+794.587847979"
Sep 11 13:32:52.340992 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:52.340909    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:52.340992 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:52.340973    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:52.353483 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:52.353459    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:32:53.121604 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:32:53.121573    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn"
Sep 11 13:34:57.199823 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.199787    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:34:57.202244 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.202207    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.204658 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.204618    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-94126e\""
Sep 11 13:34:57.204784 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.204713    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-94126e-dockercfg-pzbbx\""
Sep 11 13:34:57.217652 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.217601    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:34:57.225575 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.225550    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:34:57.227832 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.227814    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.243177 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.243151    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:34:57.318468 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.318434    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.318661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.318474    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qwtgt\" (UniqueName: \"kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.318661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.318505    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.318661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.318623    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvtmc\" (UniqueName: \"kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.419366 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.419332    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.419571 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.419372    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qwtgt\" (UniqueName: \"kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.419571 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.419399    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.419571 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.419528    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvtmc\" (UniqueName: \"kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.420106 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.420084    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.420206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.420189    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.429998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.429976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvtmc\" (UniqueName: \"kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc\") pod \"payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.432908 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.432868    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qwtgt\" (UniqueName: \"kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt\") pod \"payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") " pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.512740 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.512710    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:57.537418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.537386    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:57.801412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.801388    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:34:57.803255 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:34:57.803219    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedbe8644_c1d7_4e70_af3e_3a4adc85156b.slice/crio-8ecb06a49d9742eaced9cf9f90a3d860c5c3a1f62101629740a3439b5ecc2118 WatchSource:0}: Error finding container 8ecb06a49d9742eaced9cf9f90a3d860c5c3a1f62101629740a3439b5ecc2118: Status 404 returned error can't find the container with id 8ecb06a49d9742eaced9cf9f90a3d860c5c3a1f62101629740a3439b5ecc2118
Sep 11 13:34:57.846286 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:57.846216    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:34:57.851595 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:34:57.851564    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode35498b9_ace4_40b5_8580_6c6a7caff743.slice/crio-b963b3e1b2b091c29db9ecbefa00b7a0ff0eeb04654a4bf21987393a41237f37 WatchSource:0}: Error finding container b963b3e1b2b091c29db9ecbefa00b7a0ff0eeb04654a4bf21987393a41237f37: Status 404 returned error can't find the container with id b963b3e1b2b091c29db9ecbefa00b7a0ff0eeb04654a4bf21987393a41237f37
Sep 11 13:34:58.658853 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.658818    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" event={"ID":"e35498b9-ace4-40b5-8580-6c6a7caff743","Type":"ContainerStarted","Data":"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"}
Sep 11 13:34:58.659315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.658870    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" event={"ID":"e35498b9-ace4-40b5-8580-6c6a7caff743","Type":"ContainerStarted","Data":"b963b3e1b2b091c29db9ecbefa00b7a0ff0eeb04654a4bf21987393a41237f37"}
Sep 11 13:34:58.659315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.658914    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:34:58.660537 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.660513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" event={"ID":"edbe8644-c1d7-4e70-af3e-3a4adc85156b","Type":"ContainerStarted","Data":"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"}
Sep 11 13:34:58.660655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.660560    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" event={"ID":"edbe8644-c1d7-4e70-af3e-3a4adc85156b","Type":"ContainerStarted","Data":"8ecb06a49d9742eaced9cf9f90a3d860c5c3a1f62101629740a3439b5ecc2118"}
Sep 11 13:34:58.660655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.660582    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:34:58.680279 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.680228    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" podStartSLOduration=1.680209005 podStartE2EDuration="1.680209005s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:34:58.679326872 +0000 UTC m=+930.180231340" watchObservedRunningTime="2026-09-11 13:34:58.680209005 +0000 UTC m=+930.181113464"
Sep 11 13:34:58.700248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:34:58.700195    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" podStartSLOduration=1.700175829 podStartE2EDuration="1.700175829s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:34:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:34:58.698794762 +0000 UTC m=+930.199699235" watchObservedRunningTime="2026-09-11 13:34:58.700175829 +0000 UTC m=+930.201080279"
Sep 11 13:35:09.669144 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.669110    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:35:09.671483 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.669299    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:35:09.940403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.940326    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:35:09.943463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.943443    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:09.945580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.945563    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-autores-a61c70\""
Sep 11 13:35:09.945948 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.945931    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-autores-a61c70-dockercfg-dh24h\""
Sep 11 13:35:09.956280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:09.956258    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:35:10.038197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.038162    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.038197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.038199    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gh8mc\" (UniqueName: \"kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.053463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.053428    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:35:10.057004 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.056980    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.068085 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.068057    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:35:10.139364 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.139325    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.139543 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.139374    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z24xh\" (UniqueName: \"kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.139543 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.139407    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.139543 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.139425    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gh8mc\" (UniqueName: \"kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.140154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.140136    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.149284 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.149257    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-gh8mc\" (UniqueName: \"kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc\") pod \"payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") " pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.240464 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.240423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.240648 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.240490    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z24xh\" (UniqueName: \"kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.241132 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.241113    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.256189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.256160    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.263288 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.263258    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-z24xh\" (UniqueName: \"kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh\") pod \"payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") " pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.368241 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.368194    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:10.547353 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.547289    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:35:10.550182 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:10.550121    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5f24e9f_9009_420a_80b8_1f7e5a4de630.slice/crio-0a6c8f556ba23cc3571499223c462435e607913e63f7c39d462192b68910a5f3 WatchSource:0}: Error finding container 0a6c8f556ba23cc3571499223c462435e607913e63f7c39d462192b68910a5f3: Status 404 returned error can't find the container with id 0a6c8f556ba23cc3571499223c462435e607913e63f7c39d462192b68910a5f3
Sep 11 13:35:10.653452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.653383    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:35:10.656838 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:10.656798    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod918732ca_85ce_47d8_ab21_57bd06cdaf46.slice/crio-3af9a746fa5c0bdb3454a8b25b5662e59813e6e1169935ebca5985a6f906235e WatchSource:0}: Error finding container 3af9a746fa5c0bdb3454a8b25b5662e59813e6e1169935ebca5985a6f906235e: Status 404 returned error can't find the container with id 3af9a746fa5c0bdb3454a8b25b5662e59813e6e1169935ebca5985a6f906235e
Sep 11 13:35:10.717529 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.717489    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" event={"ID":"918732ca-85ce-47d8-ab21-57bd06cdaf46","Type":"ContainerStarted","Data":"3af9a746fa5c0bdb3454a8b25b5662e59813e6e1169935ebca5985a6f906235e"}
Sep 11 13:35:10.719302 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.719268    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" event={"ID":"d5f24e9f-9009-420a-80b8-1f7e5a4de630","Type":"ContainerStarted","Data":"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"}
Sep 11 13:35:10.719450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.719325    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" event={"ID":"d5f24e9f-9009-420a-80b8-1f7e5a4de630","Type":"ContainerStarted","Data":"0a6c8f556ba23cc3571499223c462435e607913e63f7c39d462192b68910a5f3"}
Sep 11 13:35:10.719450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.719425    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:10.749678 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:10.749598    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" podStartSLOduration=1.749576314 podStartE2EDuration="1.749576314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:10.748753273 +0000 UTC m=+942.249657742" watchObservedRunningTime="2026-09-11 13:35:10.749576314 +0000 UTC m=+942.250480779"
Sep 11 13:35:11.726780 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:11.726725    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" event={"ID":"918732ca-85ce-47d8-ab21-57bd06cdaf46","Type":"ContainerStarted","Data":"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"}
Sep 11 13:35:11.727239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:11.726808    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:11.862999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:11.862946    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" podStartSLOduration=1.862926185 podStartE2EDuration="1.862926185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:11.858322543 +0000 UTC m=+943.359226991" watchObservedRunningTime="2026-09-11 13:35:11.862926185 +0000 UTC m=+943.363830634"
Sep 11 13:35:12.033197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.033115    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:12.036592 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.036574    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.050852 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.050826    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-5305d131\""
Sep 11 13:35:12.051338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.051307    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:12.159572 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.159536    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g986t\" (UniqueName: \"kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.159766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.159580    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.248864 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.248829    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:12.252666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.252646    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.255381 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.255353    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-5305d131-dockercfg-wzgwp\""
Sep 11 13:35:12.260908 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.260885    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g986t\" (UniqueName: \"kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.261012 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.260922    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.261546 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.261528    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.275535 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.275514    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:12.292800 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.292744    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-g986t\" (UniqueName: \"kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t\") pod \"payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.349316 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.349279    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.361711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.361680    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.361809 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.361722    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-84d8h\" (UniqueName: \"kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.462698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.462664    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.462698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.462700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-84d8h\" (UniqueName: \"kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.463381 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.463363    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.474346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.474313    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-84d8h\" (UniqueName: \"kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h\") pod \"payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") " pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.562918 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.562820    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:12.659441 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.659410    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:12.660700 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:12.660679    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod905b94cd_d25b_4988_beac_2768c48cb836.slice/crio-40a77feab5287c4d97f1147dd8b8df01922e12cc3cfd7aa8430fde089d115753 WatchSource:0}: Error finding container 40a77feab5287c4d97f1147dd8b8df01922e12cc3cfd7aa8430fde089d115753: Status 404 returned error can't find the container with id 40a77feab5287c4d97f1147dd8b8df01922e12cc3cfd7aa8430fde089d115753
Sep 11 13:35:12.733313 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.733278    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" event={"ID":"905b94cd-d25b-4988-beac-2768c48cb836","Type":"ContainerStarted","Data":"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"}
Sep 11 13:35:12.733758 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.733339    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" event={"ID":"905b94cd-d25b-4988-beac-2768c48cb836","Type":"ContainerStarted","Data":"40a77feab5287c4d97f1147dd8b8df01922e12cc3cfd7aa8430fde089d115753"}
Sep 11 13:35:12.733758 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.733593    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:12.778098 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.778035    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" podStartSLOduration=1.778017167 podStartE2EDuration="1.778017167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:12.777822484 +0000 UTC m=+944.278726934" watchObservedRunningTime="2026-09-11 13:35:12.778017167 +0000 UTC m=+944.278921616"
Sep 11 13:35:12.852999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:12.852974    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:12.858051 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:12.858015    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cadf1b3_2b9f_41b0_a259_9109f3e531f3.slice/crio-c6e953a1fcb76e684a3f5d5f68663f77317eeeb8cf70062226f7ee2b19ec5d8c WatchSource:0}: Error finding container c6e953a1fcb76e684a3f5d5f68663f77317eeeb8cf70062226f7ee2b19ec5d8c: Status 404 returned error can't find the container with id c6e953a1fcb76e684a3f5d5f68663f77317eeeb8cf70062226f7ee2b19ec5d8c
Sep 11 13:35:13.740372 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:13.740334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" event={"ID":"8cadf1b3-2b9f-41b0-a259-9109f3e531f3","Type":"ContainerStarted","Data":"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"}
Sep 11 13:35:13.740760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:13.740393    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" event={"ID":"8cadf1b3-2b9f-41b0-a259-9109f3e531f3","Type":"ContainerStarted","Data":"c6e953a1fcb76e684a3f5d5f68663f77317eeeb8cf70062226f7ee2b19ec5d8c"}
Sep 11 13:35:13.768416 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:13.768363    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" podStartSLOduration=1.768344559 podStartE2EDuration="1.768344559s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:13.766603267 +0000 UTC m=+945.267507736" watchObservedRunningTime="2026-09-11 13:35:13.768344559 +0000 UTC m=+945.269249007"
Sep 11 13:35:14.745369 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:14.745343    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:20.723782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.723745    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:35:20.727485 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.727464    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.730407 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.730384    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-bfc5af\""
Sep 11 13:35:20.730717 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.730700    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-bfc5af-dockercfg-9vchr\""
Sep 11 13:35:20.740783 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.740758    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:35:20.788163 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.788131    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:35:20.791397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.791378    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:20.808027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.808000    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:35:20.839028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.838981    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.839218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.839082    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6dbd\" (UniqueName: \"kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.940440 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.940399    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-szlh4\" (UniqueName: \"kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:20.940606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.940489    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.940606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.940534    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6dbd\" (UniqueName: \"kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.940606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.940562    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:20.941200 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.941179    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:20.955782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:20.955746    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6dbd\" (UniqueName: \"kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd\") pod \"payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:21.038578 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.038484    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:21.041921 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.041479    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.041921 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.041517    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-szlh4\" (UniqueName: \"kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.042356 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.042332    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.056482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.056456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-szlh4\" (UniqueName: \"kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4\") pod \"payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") " pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.103010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.102976    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.330555 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.330481    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:35:21.333595 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:21.333568    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96e88044_00ed_4ea2_9157_b9454711bf29.slice/crio-e06049c40c9fe46937283afc6bd5f76bf1b8e4b2a6c46d59e1c06c895dea42db WatchSource:0}: Error finding container e06049c40c9fe46937283afc6bd5f76bf1b8e4b2a6c46d59e1c06c895dea42db: Status 404 returned error can't find the container with id e06049c40c9fe46937283afc6bd5f76bf1b8e4b2a6c46d59e1c06c895dea42db
Sep 11 13:35:21.413394 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.413368    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:35:21.417229 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:21.417193    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod871f39e3_4467_4172_8680_9469517b9fc9.slice/crio-66b485a17a2e3e0a46ea2013099def0cbc3882f1419bd0b006f499f76090b6cb WatchSource:0}: Error finding container 66b485a17a2e3e0a46ea2013099def0cbc3882f1419bd0b006f499f76090b6cb: Status 404 returned error can't find the container with id 66b485a17a2e3e0a46ea2013099def0cbc3882f1419bd0b006f499f76090b6cb
Sep 11 13:35:21.729058 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.729026    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:35:21.783496 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.783421    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" event={"ID":"871f39e3-4467-4172-8680-9469517b9fc9","Type":"ContainerStarted","Data":"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"}
Sep 11 13:35:21.783496 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.783482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" event={"ID":"871f39e3-4467-4172-8680-9469517b9fc9","Type":"ContainerStarted","Data":"66b485a17a2e3e0a46ea2013099def0cbc3882f1419bd0b006f499f76090b6cb"}
Sep 11 13:35:21.783810 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.783736    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:21.785609 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.785581    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" event={"ID":"96e88044-00ed-4ea2-9157-b9454711bf29","Type":"ContainerStarted","Data":"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"}
Sep 11 13:35:21.785770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.785642    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" event={"ID":"96e88044-00ed-4ea2-9157-b9454711bf29","Type":"ContainerStarted","Data":"e06049c40c9fe46937283afc6bd5f76bf1b8e4b2a6c46d59e1c06c895dea42db"}
Sep 11 13:35:21.785770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.785675    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:21.813980 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.813931    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" podStartSLOduration=1.813914746 podStartE2EDuration="1.813914746s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:21.811482092 +0000 UTC m=+953.312386537" watchObservedRunningTime="2026-09-11 13:35:21.813914746 +0000 UTC m=+953.314819192"
Sep 11 13:35:21.872973 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:21.872917    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" podStartSLOduration=1.872901303 podStartE2EDuration="1.872901303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:21.87134159 +0000 UTC m=+953.372246086" watchObservedRunningTime="2026-09-11 13:35:21.872901303 +0000 UTC m=+953.373805805"
Sep 11 13:35:22.736157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:22.736129    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:35:23.742387 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:23.742357    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:25.753605 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:25.753567    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:32.067068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.067034    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"]
Sep 11 13:35:32.071737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.071721    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:32.074740 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.074718    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:35:32.074849 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.074718    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:35:32.081885 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.081860    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"]
Sep 11 13:35:32.137424 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.137388    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5305d131-f7lgr\" (UID: \"fad9393a-7c48-4033-ac35-a756577ba4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:32.238231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.238191    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5305d131-f7lgr\" (UID: \"fad9393a-7c48-4033-ac35-a756577ba4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:32.238865 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.238845    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-5305d131-f7lgr\" (UID: \"fad9393a-7c48-4033-ac35-a756577ba4c4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:32.383527 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.383442    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:32.677793 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.677767    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"]
Sep 11 13:35:32.680342 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:32.680315    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfad9393a_7c48_4033_ac35_a756577ba4c4.slice/crio-a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e WatchSource:0}: Error finding container a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e: Status 404 returned error can't find the container with id a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e
Sep 11 13:35:32.792839 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.792804    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:35:32.793026 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.792991    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:35:32.839712 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:32.839676    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr" event={"ID":"fad9393a-7c48-4033-ac35-a756577ba4c4","Type":"ContainerStarted","Data":"a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e"}
Sep 11 13:35:37.868072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:37.868037    2551 generic.go:356] "Generic (PLEG): container finished" podID="fad9393a-7c48-4033-ac35-a756577ba4c4" containerID="616b386d4dcf26521741e08fe21a3c2a732b441367339c20ce63e942b17a0ee3" exitCode=0
Sep 11 13:35:37.868437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:37.868128    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr" event={"ID":"fad9393a-7c48-4033-ac35-a756577ba4c4","Type":"ContainerDied","Data":"616b386d4dcf26521741e08fe21a3c2a732b441367339c20ce63e942b17a0ee3"}
Sep 11 13:35:39.008953 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.008929    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:39.090877 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.090840    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca\") pod \"fad9393a-7c48-4033-ac35-a756577ba4c4\" (UID: \"fad9393a-7c48-4033-ac35-a756577ba4c4\") "
Sep 11 13:35:39.091250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.091228    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "fad9393a-7c48-4033-ac35-a756577ba4c4" (UID: "fad9393a-7c48-4033-ac35-a756577ba4c4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:35:39.191994 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.191915    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/fad9393a-7c48-4033-ac35-a756577ba4c4-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:35:39.879622 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.879594    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"
Sep 11 13:35:39.879622 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.879601    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr" event={"ID":"fad9393a-7c48-4033-ac35-a756577ba4c4","Type":"ContainerDied","Data":"a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e"}
Sep 11 13:35:39.879951 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:39.879666    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a4360630816daae6994e493e50420e78d7636575152fb681274116c8051dd57e"
Sep 11 13:35:49.074441 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.074398    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:49.075036 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.074981    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" podUID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" containerName="payload-processing" containerID="cri-o://bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f" gracePeriod=30
Sep 11 13:35:49.090864 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.090839    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:49.091092 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.091047    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" podUID="905b94cd-d25b-4988-beac-2768c48cb836" containerName="payload-pre-processing" containerID="cri-o://65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc" gracePeriod=30
Sep 11 13:35:49.330212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.330181    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"]
Sep 11 13:35:49.333356 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.333333    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-5305d131-f7lgr"]
Sep 11 13:35:49.350877 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.350853    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:49.375385 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.375238    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-84d8h\" (UniqueName: \"kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h\") pod \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") "
Sep 11 13:35:49.375385 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.375320    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume\") pod \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\" (UID: \"8cadf1b3-2b9f-41b0-a259-9109f3e531f3\") "
Sep 11 13:35:49.375934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.375906    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "8cadf1b3-2b9f-41b0-a259-9109f3e531f3" (UID: "8cadf1b3-2b9f-41b0-a259-9109f3e531f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:35:49.377759 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.377730    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h" (OuterVolumeSpecName: "kube-api-access-84d8h") pod "8cadf1b3-2b9f-41b0-a259-9109f3e531f3" (UID: "8cadf1b3-2b9f-41b0-a259-9109f3e531f3"). InnerVolumeSpecName "kube-api-access-84d8h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:35:49.378269 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.378249    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:49.476262 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.476229    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume\") pod \"905b94cd-d25b-4988-beac-2768c48cb836\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") "
Sep 11 13:35:49.476424 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.476347    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-g986t\" (UniqueName: \"kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t\") pod \"905b94cd-d25b-4988-beac-2768c48cb836\" (UID: \"905b94cd-d25b-4988-beac-2768c48cb836\") "
Sep 11 13:35:49.476540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.476525    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-84d8h\" (UniqueName: \"kubernetes.io/projected/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-kube-api-access-84d8h\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:35:49.476607 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.476544    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8cadf1b3-2b9f-41b0-a259-9109f3e531f3-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:35:49.476688 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.476605    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "905b94cd-d25b-4988-beac-2768c48cb836" (UID: "905b94cd-d25b-4988-beac-2768c48cb836"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:35:49.478736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.478713    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t" (OuterVolumeSpecName: "kube-api-access-g986t") pod "905b94cd-d25b-4988-beac-2768c48cb836" (UID: "905b94cd-d25b-4988-beac-2768c48cb836"). InnerVolumeSpecName "kube-api-access-g986t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:35:49.577317 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.577277    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g986t\" (UniqueName: \"kubernetes.io/projected/905b94cd-d25b-4988-beac-2768c48cb836-kube-api-access-g986t\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:35:49.577317 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.577310    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/905b94cd-d25b-4988-beac-2768c48cb836-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:35:49.928860 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.928829    2551 generic.go:356] "Generic (PLEG): container finished" podID="905b94cd-d25b-4988-beac-2768c48cb836" containerID="65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc" exitCode=0
Sep 11 13:35:49.929061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.928873    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"
Sep 11 13:35:49.929061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.928907    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" event={"ID":"905b94cd-d25b-4988-beac-2768c48cb836","Type":"ContainerDied","Data":"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"}
Sep 11 13:35:49.929061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.928983    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq" event={"ID":"905b94cd-d25b-4988-beac-2768c48cb836","Type":"ContainerDied","Data":"40a77feab5287c4d97f1147dd8b8df01922e12cc3cfd7aa8430fde089d115753"}
Sep 11 13:35:49.929061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.929008    2551 scope.go:117] "RemoveContainer" containerID="65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"
Sep 11 13:35:49.930405 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.930381    2551 generic.go:356] "Generic (PLEG): container finished" podID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" containerID="bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f" exitCode=0
Sep 11 13:35:49.930512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.930411    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" event={"ID":"8cadf1b3-2b9f-41b0-a259-9109f3e531f3","Type":"ContainerDied","Data":"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"}
Sep 11 13:35:49.930512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.930434    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"
Sep 11 13:35:49.930512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.930454    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8" event={"ID":"8cadf1b3-2b9f-41b0-a259-9109f3e531f3","Type":"ContainerDied","Data":"c6e953a1fcb76e684a3f5d5f68663f77317eeeb8cf70062226f7ee2b19ec5d8c"}
Sep 11 13:35:49.939379 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.939361    2551 scope.go:117] "RemoveContainer" containerID="65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"
Sep 11 13:35:49.939705 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:35:49.939681    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc\": container with ID starting with 65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc not found: ID does not exist" containerID="65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"
Sep 11 13:35:49.939800 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.939716    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc"} err="failed to get container status \"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc\": rpc error: code = NotFound desc = could not find container \"65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc\": container with ID starting with 65520f6e9f2f999bc0dc3ad1cdf20d483bee11e5adb34722fb385d2f383f31cc not found: ID does not exist"
Sep 11 13:35:49.939800 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.939740    2551 scope.go:117] "RemoveContainer" containerID="bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"
Sep 11 13:35:49.948236 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.948217    2551 scope.go:117] "RemoveContainer" containerID="bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"
Sep 11 13:35:49.948560 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:35:49.948537    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f\": container with ID starting with bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f not found: ID does not exist" containerID="bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"
Sep 11 13:35:49.948692 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.948564    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f"} err="failed to get container status \"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f\": rpc error: code = NotFound desc = could not find container \"bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f\": container with ID starting with bbd4f3d18154f186eee7ad9fe0669cc13e580f9485328724372e4e798b92ba4f not found: ID does not exist"
Sep 11 13:35:49.960686 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.960662    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:49.969262 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:49.969240    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-5305d131-56f5cbf774-prnwq"]
Sep 11 13:35:50.012329 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:50.012295    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:50.018413 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:50.018389    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-5305d131-7fb6756df5-7bzl8"]
Sep 11 13:35:50.963510 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:50.963479    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" path="/var/lib/kubelet/pods/8cadf1b3-2b9f-41b0-a259-9109f3e531f3/volumes"
Sep 11 13:35:50.963912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:50.963876    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="905b94cd-d25b-4988-beac-2768c48cb836" path="/var/lib/kubelet/pods/905b94cd-d25b-4988-beac-2768c48cb836/volumes"
Sep 11 13:35:50.964158 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:50.964145    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fad9393a-7c48-4033-ac35-a756577ba4c4" path="/var/lib/kubelet/pods/fad9393a-7c48-4033-ac35-a756577ba4c4/volumes"
Sep 11 13:35:55.665441 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665401    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:35:55.665923 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665905    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="905b94cd-d25b-4988-beac-2768c48cb836" containerName="payload-pre-processing"
Sep 11 13:35:55.665979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665926    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="905b94cd-d25b-4988-beac-2768c48cb836" containerName="payload-pre-processing"
Sep 11 13:35:55.665979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665938    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" containerName="payload-processing"
Sep 11 13:35:55.665979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665944    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" containerName="payload-processing"
Sep 11 13:35:55.665979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665954    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fad9393a-7c48-4033-ac35-a756577ba4c4" containerName="revoke-keys"
Sep 11 13:35:55.665979 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.665959    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fad9393a-7c48-4033-ac35-a756577ba4c4" containerName="revoke-keys"
Sep 11 13:35:55.666130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.666009    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="905b94cd-d25b-4988-beac-2768c48cb836" containerName="payload-pre-processing"
Sep 11 13:35:55.666130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.666018    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fad9393a-7c48-4033-ac35-a756577ba4c4" containerName="revoke-keys"
Sep 11 13:35:55.666130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.666029    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8cadf1b3-2b9f-41b0-a259-9109f3e531f3" containerName="payload-processing"
Sep 11 13:35:55.668891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.668876    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.672226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.672204    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-migrate-9acc1188\""
Sep 11 13:35:55.702813 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.702783    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:35:55.725881 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.725849    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:35:55.729728 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.729709    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.732143 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.732121    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.732256 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.732150    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qfkqw\" (UniqueName: \"kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.735691 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.735667    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-migrate-9acc1188-dockercfg-d4pr7\""
Sep 11 13:35:55.750771 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.750750    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:35:55.833417 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.833380    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.833588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.833452    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9qmvx\" (UniqueName: \"kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.833588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.833502    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.833588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.833523    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qfkqw\" (UniqueName: \"kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.834212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.834191    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.842848 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.842815    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qfkqw\" (UniqueName: \"kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw\") pod \"payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") " pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:55.934336 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.934253    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9qmvx\" (UniqueName: \"kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.934336 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.934304    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.934970 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.934953    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.954463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.954431    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9qmvx\" (UniqueName: \"kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx\") pod \"payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") " pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:55.979866 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:55.979834    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:56.040195 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.039983    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:56.292704 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:56.292505    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode092dd9c_19ee_4cc8_912c_9dd9aa2dfcec.slice/crio-e90d6ab0eda5cc04ee30c9ff3fbbf210773c23a1bd67ab7a57eed7a0180d6df7 WatchSource:0}: Error finding container e90d6ab0eda5cc04ee30c9ff3fbbf210773c23a1bd67ab7a57eed7a0180d6df7: Status 404 returned error can't find the container with id e90d6ab0eda5cc04ee30c9ff3fbbf210773c23a1bd67ab7a57eed7a0180d6df7
Sep 11 13:35:56.294573 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.294550    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:35:56.340417 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.340390    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:35:56.341038 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:35:56.341014    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod01fc0451_9366_4142_9257_28d0387e8ad8.slice/crio-b50f2876f752cc463c8fe31a0d69ab834ac8a828bce93775f905e7bc7e8245e0 WatchSource:0}: Error finding container b50f2876f752cc463c8fe31a0d69ab834ac8a828bce93775f905e7bc7e8245e0: Status 404 returned error can't find the container with id b50f2876f752cc463c8fe31a0d69ab834ac8a828bce93775f905e7bc7e8245e0
Sep 11 13:35:56.967877 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.967842    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" event={"ID":"01fc0451-9366-4142-9257-28d0387e8ad8","Type":"ContainerStarted","Data":"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"}
Sep 11 13:35:56.968343 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.968008    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:35:56.968343 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.968060    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" event={"ID":"01fc0451-9366-4142-9257-28d0387e8ad8","Type":"ContainerStarted","Data":"b50f2876f752cc463c8fe31a0d69ab834ac8a828bce93775f905e7bc7e8245e0"}
Sep 11 13:35:56.969661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.969613    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" event={"ID":"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec","Type":"ContainerStarted","Data":"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"}
Sep 11 13:35:56.969792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.969676    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:35:56.969792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.969701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" event={"ID":"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec","Type":"ContainerStarted","Data":"e90d6ab0eda5cc04ee30c9ff3fbbf210773c23a1bd67ab7a57eed7a0180d6df7"}
Sep 11 13:35:56.987851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:56.987791    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" podStartSLOduration=1.987775106 podStartE2EDuration="1.987775106s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:56.98631965 +0000 UTC m=+988.487224111" watchObservedRunningTime="2026-09-11 13:35:56.987775106 +0000 UTC m=+988.488679555"
Sep 11 13:35:57.003866 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:35:57.003822    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" podStartSLOduration=2.003807833 podStartE2EDuration="2.003807833s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:35:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:35:57.003038456 +0000 UTC m=+988.503942907" watchObservedRunningTime="2026-09-11 13:35:57.003807833 +0000 UTC m=+988.504712281"
Sep 11 13:36:07.977836 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:07.977803    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:36:07.978203 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:07.977886    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:36:18.964101 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:18.964070    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"]
Sep 11 13:36:18.971023 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:18.971005    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:18.974141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:18.974119    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:36:18.974282 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:18.974124    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:36:18.980805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:18.980778    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"]
Sep 11 13:36:19.040382 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:19.040338    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5\" (UID: \"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:19.141389 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:19.141354    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5\" (UID: \"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:19.142021 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:19.142003    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5\" (UID: \"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:19.283244 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:19.283207    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:19.556292 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:19.556208    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"]
Sep 11 13:36:19.559242 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:36:19.559210    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ca5d9e0_dfcb_49fa_98aa_8fca5a25764e.slice/crio-8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf WatchSource:0}: Error finding container 8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf: Status 404 returned error can't find the container with id 8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf
Sep 11 13:36:20.088216 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:20.088175    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5" event={"ID":"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e","Type":"ContainerStarted","Data":"9c980da8e71ffe1701eeaafcfdcbae4f844977315b377feeede7496116dd6c61"}
Sep 11 13:36:20.088653 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:20.088242    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5" event={"ID":"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e","Type":"ContainerStarted","Data":"8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf"}
Sep 11 13:36:20.103074 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:20.103019    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5" podStartSLOduration=2.102999791 podStartE2EDuration="2.102999791s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:36:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:36:20.102797772 +0000 UTC m=+1011.603702256" watchObservedRunningTime="2026-09-11 13:36:20.102999791 +0000 UTC m=+1011.603904255"
Sep 11 13:36:21.095250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:21.095220    2551 generic.go:356] "Generic (PLEG): container finished" podID="4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" containerID="9c980da8e71ffe1701eeaafcfdcbae4f844977315b377feeede7496116dd6c61" exitCode=0
Sep 11 13:36:21.095673 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:21.095305    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5" event={"ID":"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e","Type":"ContainerDied","Data":"9c980da8e71ffe1701eeaafcfdcbae4f844977315b377feeede7496116dd6c61"}
Sep 11 13:36:22.229029 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:22.229004    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:22.371299 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:22.371210    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca\") pod \"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e\" (UID: \"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e\") "
Sep 11 13:36:22.371653 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:22.371595    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" (UID: "4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:36:22.472694 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:22.472652    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:23.106401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:23.106362    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5" event={"ID":"4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e","Type":"ContainerDied","Data":"8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf"}
Sep 11 13:36:23.106401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:23.106393    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"
Sep 11 13:36:23.106597 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:23.106422    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8b57600028326f6fa8fb76e8b74b80e4d41278386b1f2bda880b0ff6b297f2cf"
Sep 11 13:36:25.328668 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.328608    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:36:25.329206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.329025    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" containerName="revoke-keys"
Sep 11 13:36:25.329206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.329038    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" containerName="revoke-keys"
Sep 11 13:36:25.329206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.329094    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" containerName="revoke-keys"
Sep 11 13:36:25.332197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.332181    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.335890 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.335867    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-94126e\"/\"default-dockercfg-ncpsb\""
Sep 11 13:36:25.336129 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.336114    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-a-94126e\"/\"test-model-94126e-kserve-self-signed-certs\""
Sep 11 13:36:25.336181 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.336117    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-94126e\"/\"openshift-service-ca.crt\""
Sep 11 13:36:25.336264 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.336247    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-a-94126e\"/\"kube-root-ca.crt\""
Sep 11 13:36:25.343304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.343284    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:36:25.501304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501267    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.501304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501304    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.501501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501324    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.501501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501368    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4rfj5\" (UniqueName: \"kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.501501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501442    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.501501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.501467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602342    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602384    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602427 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602423    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602438    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602454    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.602731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602479    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4rfj5\" (UniqueName: \"kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.603013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.602991    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.603089 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.603045    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.603142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.603086    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.604807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.604785    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.605055 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.605035    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.624514 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.624491    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-4rfj5\" (UniqueName: \"kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5\") pod \"test-model-94126e-kserve-785769d4d7-tsn6v\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") " pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.641360 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.641328    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:25.923013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:25.922981    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:36:25.924090 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:36:25.924062    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0f5d9c6e_9caf_4425_ad44_960b9d1326b6.slice/crio-6fd6a79ef90f2afc951cef8db1ef4bcc500298ae6178702b4cacf8dad28a44d4 WatchSource:0}: Error finding container 6fd6a79ef90f2afc951cef8db1ef4bcc500298ae6178702b4cacf8dad28a44d4: Status 404 returned error can't find the container with id 6fd6a79ef90f2afc951cef8db1ef4bcc500298ae6178702b4cacf8dad28a44d4
Sep 11 13:36:26.124717 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.124677    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" event={"ID":"0f5d9c6e-9caf-4425-ad44-960b9d1326b6","Type":"ContainerStarted","Data":"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"}
Sep 11 13:36:26.124883 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.124732    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" event={"ID":"0f5d9c6e-9caf-4425-ad44-960b9d1326b6","Type":"ContainerStarted","Data":"6fd6a79ef90f2afc951cef8db1ef4bcc500298ae6178702b4cacf8dad28a44d4"}
Sep 11 13:36:26.150384 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.150317    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" podStartSLOduration=1.150293608 podStartE2EDuration="1.150293608s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:36:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:36:26.146663924 +0000 UTC m=+1017.647568402" watchObservedRunningTime="2026-09-11 13:36:26.150293608 +0000 UTC m=+1017.651198058"
Sep 11 13:36:26.228027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.227997    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:36:26.228318 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.228284    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" podUID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" containerName="payload-processing" containerID="cri-o://274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd" gracePeriod=30
Sep 11 13:36:26.243944 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.243915    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:36:26.244143 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.244121    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" podUID="01fc0451-9366-4142-9257-28d0387e8ad8" containerName="payload-pre-processing" containerID="cri-o://ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0" gracePeriod=30
Sep 11 13:36:26.476201 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.476173    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:36:26.514562 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.514537    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:36:26.609964 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.609933    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9qmvx\" (UniqueName: \"kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx\") pod \"01fc0451-9366-4142-9257-28d0387e8ad8\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") "
Sep 11 13:36:26.610146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.609983    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume\") pod \"01fc0451-9366-4142-9257-28d0387e8ad8\" (UID: \"01fc0451-9366-4142-9257-28d0387e8ad8\") "
Sep 11 13:36:26.610146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.610022    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume\") pod \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") "
Sep 11 13:36:26.610146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.610056    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qfkqw\" (UniqueName: \"kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw\") pod \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\" (UID: \"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec\") "
Sep 11 13:36:26.610437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.610408    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "01fc0451-9366-4142-9257-28d0387e8ad8" (UID: "01fc0451-9366-4142-9257-28d0387e8ad8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:36:26.610564 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.610436    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" (UID: "e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:36:26.612548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.612207    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx" (OuterVolumeSpecName: "kube-api-access-9qmvx") pod "01fc0451-9366-4142-9257-28d0387e8ad8" (UID: "01fc0451-9366-4142-9257-28d0387e8ad8"). InnerVolumeSpecName "kube-api-access-9qmvx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:36:26.612548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.612230    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw" (OuterVolumeSpecName: "kube-api-access-qfkqw") pod "e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" (UID: "e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec"). InnerVolumeSpecName "kube-api-access-qfkqw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:36:26.711395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.711355    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/01fc0451-9366-4142-9257-28d0387e8ad8-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:26.711395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.711391    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:26.711395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.711401    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qfkqw\" (UniqueName: \"kubernetes.io/projected/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec-kube-api-access-qfkqw\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:26.711395 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.711410    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9qmvx\" (UniqueName: \"kubernetes.io/projected/01fc0451-9366-4142-9257-28d0387e8ad8-kube-api-access-9qmvx\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:26.725446 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.725413    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"]
Sep 11 13:36:26.729428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.729404    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-9acc1188-nrlz5"]
Sep 11 13:36:26.964828 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:26.964749    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e" path="/var/lib/kubelet/pods/4ca5d9e0-dfcb-49fa-98aa-8fca5a25764e/volumes"
Sep 11 13:36:27.130609 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.130577    2551 generic.go:356] "Generic (PLEG): container finished" podID="01fc0451-9366-4142-9257-28d0387e8ad8" containerID="ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0" exitCode=0
Sep 11 13:36:27.130782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.130637    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"
Sep 11 13:36:27.130782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.130667    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" event={"ID":"01fc0451-9366-4142-9257-28d0387e8ad8","Type":"ContainerDied","Data":"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"}
Sep 11 13:36:27.130782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.130738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck" event={"ID":"01fc0451-9366-4142-9257-28d0387e8ad8","Type":"ContainerDied","Data":"b50f2876f752cc463c8fe31a0d69ab834ac8a828bce93775f905e7bc7e8245e0"}
Sep 11 13:36:27.130782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.130773    2551 scope.go:117] "RemoveContainer" containerID="ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"
Sep 11 13:36:27.132250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.132143    2551 generic.go:356] "Generic (PLEG): container finished" podID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" containerID="274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd" exitCode=0
Sep 11 13:36:27.132250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.132193    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"
Sep 11 13:36:27.132347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.132190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" event={"ID":"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec","Type":"ContainerDied","Data":"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"}
Sep 11 13:36:27.132347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.132316    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp" event={"ID":"e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec","Type":"ContainerDied","Data":"e90d6ab0eda5cc04ee30c9ff3fbbf210773c23a1bd67ab7a57eed7a0180d6df7"}
Sep 11 13:36:27.139669 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.139622    2551 scope.go:117] "RemoveContainer" containerID="ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"
Sep 11 13:36:27.139910 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:36:27.139889    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0\": container with ID starting with ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0 not found: ID does not exist" containerID="ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"
Sep 11 13:36:27.140017 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.139916    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0"} err="failed to get container status \"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0\": rpc error: code = NotFound desc = could not find container \"ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0\": container with ID starting with ee7506bae803fde03e1f7d3a4c732d7a8087add7f43892dadaf8ef1acbb2aed0 not found: ID does not exist"
Sep 11 13:36:27.140017 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.139931    2551 scope.go:117] "RemoveContainer" containerID="274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"
Sep 11 13:36:27.147479 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.147461    2551 scope.go:117] "RemoveContainer" containerID="274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"
Sep 11 13:36:27.147797 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:36:27.147773    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd\": container with ID starting with 274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd not found: ID does not exist" containerID="274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"
Sep 11 13:36:27.147887 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.147806    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd"} err="failed to get container status \"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd\": rpc error: code = NotFound desc = could not find container \"274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd\": container with ID starting with 274b9d426ddab46f903bb137ab51bd9126e5b17d42059bac76a3ab4fd3d402dd not found: ID does not exist"
Sep 11 13:36:27.154435 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.154415    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:36:27.160155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.160133    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-migrate-9acc1188-58cdb8c8b6-9chvp"]
Sep 11 13:36:27.173212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.173187    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:36:27.183504 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:27.183480    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-migrate-9acc1188-86b88b9566-7k9ck"]
Sep 11 13:36:28.964279 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:28.964247    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="01fc0451-9366-4142-9257-28d0387e8ad8" path="/var/lib/kubelet/pods/01fc0451-9366-4142-9257-28d0387e8ad8/volumes"
Sep 11 13:36:28.964607 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:28.964575    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" path="/var/lib/kubelet/pods/e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec/volumes"
Sep 11 13:36:34.890847 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.890812    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:36:34.891234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891160    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" containerName="payload-processing"
Sep 11 13:36:34.891234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891171    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" containerName="payload-processing"
Sep 11 13:36:34.891234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891197    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="01fc0451-9366-4142-9257-28d0387e8ad8" containerName="payload-pre-processing"
Sep 11 13:36:34.891234 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891203    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="01fc0451-9366-4142-9257-28d0387e8ad8" containerName="payload-pre-processing"
Sep 11 13:36:34.891418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891258    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="01fc0451-9366-4142-9257-28d0387e8ad8" containerName="payload-pre-processing"
Sep 11 13:36:34.891418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.891266    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e092dd9c-19ee-4cc8-912c-9dd9aa2dfcec" containerName="payload-processing"
Sep 11 13:36:34.895899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.895881    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:34.898339 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.898320    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-db87970d\""
Sep 11 13:36:34.915977 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.915948    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:36:34.984859 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.984823    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:34.985021 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:34.984917    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wt7pc\" (UniqueName: \"kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.024408 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.024381    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:36:35.027649 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.027619    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.030056 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.030031    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-db87970d-dockercfg-4mrgk\""
Sep 11 13:36:35.045090 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.045065    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:36:35.085855 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.085821    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.086043 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.085869    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.086043 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.085891    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2qdq\" (UniqueName: \"kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.086156 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.086042    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wt7pc\" (UniqueName: \"kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.086706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.086685    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.107670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.107617    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-wt7pc\" (UniqueName: \"kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc\") pod \"payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.186851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.186768    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.186851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.186806    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2qdq\" (UniqueName: \"kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.187498 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.187473    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.205770 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.205741    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:35.220782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.220750    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2qdq\" (UniqueName: \"kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq\") pod \"payload-processing-e2e-ait-db87970d-54d5696d77-r5brc\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") " pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.338176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.338133    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:35.481500 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.481471    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:36:35.483079 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:36:35.483051    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod34917548_bf6d_43a8_a846_68b04dc9bcb4.slice/crio-9777750d87d5a6449c5b5cfa2b5d1b2d9ae046873bdcaf48b650ce6fdaa9edcb WatchSource:0}: Error finding container 9777750d87d5a6449c5b5cfa2b5d1b2d9ae046873bdcaf48b650ce6fdaa9edcb: Status 404 returned error can't find the container with id 9777750d87d5a6449c5b5cfa2b5d1b2d9ae046873bdcaf48b650ce6fdaa9edcb
Sep 11 13:36:35.626036 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.626002    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:36:35.629265 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:36:35.629238    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9afec799_e851_4930_b9b8_927382ec7fcf.slice/crio-14eaa5c3c4b169a43b58587ee4b830ec3fa8c3bcd48e74760cde5e7c498a548c WatchSource:0}: Error finding container 14eaa5c3c4b169a43b58587ee4b830ec3fa8c3bcd48e74760cde5e7c498a548c: Status 404 returned error can't find the container with id 14eaa5c3c4b169a43b58587ee4b830ec3fa8c3bcd48e74760cde5e7c498a548c
Sep 11 13:36:35.641521 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.641500    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:35.641575 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.641546    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:35.656864 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:35.656840    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:36.186760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.186719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" event={"ID":"34917548-bf6d-43a8-a846-68b04dc9bcb4","Type":"ContainerStarted","Data":"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"}
Sep 11 13:36:36.187243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.186787    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" event={"ID":"34917548-bf6d-43a8-a846-68b04dc9bcb4","Type":"ContainerStarted","Data":"9777750d87d5a6449c5b5cfa2b5d1b2d9ae046873bdcaf48b650ce6fdaa9edcb"}
Sep 11 13:36:36.187243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.186867    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:36.188498 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.188473    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" event={"ID":"9afec799-e851-4930-b9b8-927382ec7fcf","Type":"ContainerStarted","Data":"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"}
Sep 11 13:36:36.188604 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.188513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" event={"ID":"9afec799-e851-4930-b9b8-927382ec7fcf","Type":"ContainerStarted","Data":"14eaa5c3c4b169a43b58587ee4b830ec3fa8c3bcd48e74760cde5e7c498a548c"}
Sep 11 13:36:36.200580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.200553    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:36:36.206239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.206175    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" podStartSLOduration=2.206152917 podStartE2EDuration="2.206152917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:36:36.204327217 +0000 UTC m=+1027.705231683" watchObservedRunningTime="2026-09-11 13:36:36.206152917 +0000 UTC m=+1027.707057365"
Sep 11 13:36:36.259175 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:36.259125    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" podStartSLOduration=2.25910834 podStartE2EDuration="2.25910834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:36:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:36:36.258598347 +0000 UTC m=+1027.759502817" watchObservedRunningTime="2026-09-11 13:36:36.25910834 +0000 UTC m=+1027.760012788"
Sep 11 13:36:37.194012 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:37.193968    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:47.196101 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:47.196066    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:36:48.201173 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:48.201144    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:36:55.252665 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.252614    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"]
Sep 11 13:36:55.255935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.255919    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:55.261110 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.261082    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:36:55.264057 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.264039    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:36:55.280653 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.280612    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"]
Sep 11 13:36:55.348913 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.348871    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a61c70-92mm2\" (UID: \"4377c82e-df61-439c-b5ef-4ccb81f3ce9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:55.450401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.450357    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a61c70-92mm2\" (UID: \"4377c82e-df61-439c-b5ef-4ccb81f3ce9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:55.451179 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.451155    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-autores-a61c70-92mm2\" (UID: \"4377c82e-df61-439c-b5ef-4ccb81f3ce9a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:55.567168 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.567083    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:55.843161 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:55.843137    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"]
Sep 11 13:36:55.845362 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:36:55.845336    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4377c82e_df61_439c_b5ef_4ccb81f3ce9a.slice/crio-23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7 WatchSource:0}: Error finding container 23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7: Status 404 returned error can't find the container with id 23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7
Sep 11 13:36:56.286764 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:56.286729    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2" event={"ID":"4377c82e-df61-439c-b5ef-4ccb81f3ce9a","Type":"ContainerStarted","Data":"db74a6260b38722d7774475748c1a7ec02b2aba89a21b46f227b97873f1175ac"}
Sep 11 13:36:56.287124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:56.286781    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2" event={"ID":"4377c82e-df61-439c-b5ef-4ccb81f3ce9a","Type":"ContainerStarted","Data":"23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7"}
Sep 11 13:36:57.293044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:57.293004    2551 generic.go:356] "Generic (PLEG): container finished" podID="4377c82e-df61-439c-b5ef-4ccb81f3ce9a" containerID="db74a6260b38722d7774475748c1a7ec02b2aba89a21b46f227b97873f1175ac" exitCode=0
Sep 11 13:36:57.293459 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:57.293088    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2" event={"ID":"4377c82e-df61-439c-b5ef-4ccb81f3ce9a","Type":"ContainerDied","Data":"db74a6260b38722d7774475748c1a7ec02b2aba89a21b46f227b97873f1175ac"}
Sep 11 13:36:58.436261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:58.436235    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:58.477102 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:58.477072    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca\") pod \"4377c82e-df61-439c-b5ef-4ccb81f3ce9a\" (UID: \"4377c82e-df61-439c-b5ef-4ccb81f3ce9a\") "
Sep 11 13:36:58.477434 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:58.477410    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "4377c82e-df61-439c-b5ef-4ccb81f3ce9a" (UID: "4377c82e-df61-439c-b5ef-4ccb81f3ce9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:36:58.578207 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:58.578108    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/4377c82e-df61-439c-b5ef-4ccb81f3ce9a-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:36:59.305306 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:59.305227    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"
Sep 11 13:36:59.305306 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:59.305266    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2" event={"ID":"4377c82e-df61-439c-b5ef-4ccb81f3ce9a","Type":"ContainerDied","Data":"23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7"}
Sep 11 13:36:59.305495 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:36:59.305316    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="23190772ce9e7d50fdbe347e09f708d4be20eb5fff818a3f44637e0777f654f7"
Sep 11 13:37:00.915082 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.915041    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"]
Sep 11 13:37:00.915598 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.915577    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4377c82e-df61-439c-b5ef-4ccb81f3ce9a" containerName="revoke-keys"
Sep 11 13:37:00.915712 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.915600    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4377c82e-df61-439c-b5ef-4ccb81f3ce9a" containerName="revoke-keys"
Sep 11 13:37:00.915769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.915741    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4377c82e-df61-439c-b5ef-4ccb81f3ce9a" containerName="revoke-keys"
Sep 11 13:37:00.918847 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.918825    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:00.921097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.921078    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:37:00.921816 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.921800    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:37:00.927888 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.927612    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"]
Sep 11 13:37:00.995960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:00.995927    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db87970d-lrh99\" (UID: \"2458320a-d2c9-422e-a45f-c915c69a030e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:01.096886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:01.096851    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db87970d-lrh99\" (UID: \"2458320a-d2c9-422e-a45f-c915c69a030e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:01.097497 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:01.097469    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ait-db87970d-lrh99\" (UID: \"2458320a-d2c9-422e-a45f-c915c69a030e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:01.230406 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:01.230375    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:01.528202 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:01.528178    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"]
Sep 11 13:37:01.530311 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:37:01.530282    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2458320a_d2c9_422e_a45f_c915c69a030e.slice/crio-f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9 WatchSource:0}: Error finding container f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9: Status 404 returned error can't find the container with id f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9
Sep 11 13:37:02.321397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:02.321363    2551 generic.go:356] "Generic (PLEG): container finished" podID="2458320a-d2c9-422e-a45f-c915c69a030e" containerID="e638de9da3060393dfe7a73d97f90435f79adfbb8810cbd7b0a9748316dbe439" exitCode=0
Sep 11 13:37:02.321851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:02.321414    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99" event={"ID":"2458320a-d2c9-422e-a45f-c915c69a030e","Type":"ContainerDied","Data":"e638de9da3060393dfe7a73d97f90435f79adfbb8810cbd7b0a9748316dbe439"}
Sep 11 13:37:02.321851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:02.321450    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99" event={"ID":"2458320a-d2c9-422e-a45f-c915c69a030e","Type":"ContainerStarted","Data":"f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9"}
Sep 11 13:37:03.461779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:03.461752    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:03.518468 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:03.518436    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca\") pod \"2458320a-d2c9-422e-a45f-c915c69a030e\" (UID: \"2458320a-d2c9-422e-a45f-c915c69a030e\") "
Sep 11 13:37:03.518835 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:03.518811    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2458320a-d2c9-422e-a45f-c915c69a030e" (UID: "2458320a-d2c9-422e-a45f-c915c69a030e"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:03.619744 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:03.619657    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2458320a-d2c9-422e-a45f-c915c69a030e-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:04.334704 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:04.334670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99" event={"ID":"2458320a-d2c9-422e-a45f-c915c69a030e","Type":"ContainerDied","Data":"f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9"}
Sep 11 13:37:04.334704 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:04.334701    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"
Sep 11 13:37:04.334900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:04.334720    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f56a4cd54b728ea6d9be83ba9016a43becb36e8621549a04034408282993e9f9"
Sep 11 13:37:06.022225 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.022192    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:37:06.023008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.022972    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" podUID="918732ca-85ce-47d8-ab21-57bd06cdaf46" containerName="payload-processing" containerID="cri-o://c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9" gracePeriod=30
Sep 11 13:37:06.031128 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.031055    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:37:06.031570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.031276    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" podUID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" containerName="payload-pre-processing" containerID="cri-o://88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3" gracePeriod=30
Sep 11 13:37:06.216682 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.216646    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"]
Sep 11 13:37:06.219822 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.219796    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-autores-a61c70-92mm2"]
Sep 11 13:37:06.322763 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.322741    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:37:06.325467 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.325449    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:37:06.351685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.351465    2551 generic.go:356] "Generic (PLEG): container finished" podID="918732ca-85ce-47d8-ab21-57bd06cdaf46" containerID="c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9" exitCode=0
Sep 11 13:37:06.351685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.351585    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"
Sep 11 13:37:06.351685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.351571    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" event={"ID":"918732ca-85ce-47d8-ab21-57bd06cdaf46","Type":"ContainerDied","Data":"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"}
Sep 11 13:37:06.351685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.351683    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq" event={"ID":"918732ca-85ce-47d8-ab21-57bd06cdaf46","Type":"ContainerDied","Data":"3af9a746fa5c0bdb3454a8b25b5662e59813e6e1169935ebca5985a6f906235e"}
Sep 11 13:37:06.351999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.351724    2551 scope.go:117] "RemoveContainer" containerID="c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"
Sep 11 13:37:06.353297 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.353276    2551 generic.go:356] "Generic (PLEG): container finished" podID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" containerID="88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3" exitCode=0
Sep 11 13:37:06.353568 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.353313    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" event={"ID":"d5f24e9f-9009-420a-80b8-1f7e5a4de630","Type":"ContainerDied","Data":"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"}
Sep 11 13:37:06.353568 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.353535    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m" event={"ID":"d5f24e9f-9009-420a-80b8-1f7e5a4de630","Type":"ContainerDied","Data":"0a6c8f556ba23cc3571499223c462435e607913e63f7c39d462192b68910a5f3"}
Sep 11 13:37:06.353568 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.353328    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"
Sep 11 13:37:06.364147 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.364043    2551 scope.go:117] "RemoveContainer" containerID="c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"
Sep 11 13:37:06.364548 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:37:06.364434    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9\": container with ID starting with c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9 not found: ID does not exist" containerID="c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"
Sep 11 13:37:06.364548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.364476    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9"} err="failed to get container status \"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9\": rpc error: code = NotFound desc = could not find container \"c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9\": container with ID starting with c5c0d875308673305c9bbc69891c4dd9c352bd034f170b61d91de25b016b5bd9 not found: ID does not exist"
Sep 11 13:37:06.364548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.364500    2551 scope.go:117] "RemoveContainer" containerID="88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"
Sep 11 13:37:06.373982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.373901    2551 scope.go:117] "RemoveContainer" containerID="88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"
Sep 11 13:37:06.375162 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:37:06.375137    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3\": container with ID starting with 88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3 not found: ID does not exist" containerID="88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"
Sep 11 13:37:06.375291 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.375173    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3"} err="failed to get container status \"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3\": rpc error: code = NotFound desc = could not find container \"88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3\": container with ID starting with 88f6bf871f6dc4952b8d1af3ee6d7f0a55389fedc47db037bcb5bd32c0235aa3 not found: ID does not exist"
Sep 11 13:37:06.453964 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.453929    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume\") pod \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") "
Sep 11 13:37:06.454149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.454005    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume\") pod \"918732ca-85ce-47d8-ab21-57bd06cdaf46\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") "
Sep 11 13:37:06.454149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.454032    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-z24xh\" (UniqueName: \"kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh\") pod \"918732ca-85ce-47d8-ab21-57bd06cdaf46\" (UID: \"918732ca-85ce-47d8-ab21-57bd06cdaf46\") "
Sep 11 13:37:06.454149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.454065    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-gh8mc\" (UniqueName: \"kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc\") pod \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\" (UID: \"d5f24e9f-9009-420a-80b8-1f7e5a4de630\") "
Sep 11 13:37:06.454363 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.454338    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5f24e9f-9009-420a-80b8-1f7e5a4de630" (UID: "d5f24e9f-9009-420a-80b8-1f7e5a4de630"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:06.454455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.454425    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "918732ca-85ce-47d8-ab21-57bd06cdaf46" (UID: "918732ca-85ce-47d8-ab21-57bd06cdaf46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:06.456154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.456127    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc" (OuterVolumeSpecName: "kube-api-access-gh8mc") pod "d5f24e9f-9009-420a-80b8-1f7e5a4de630" (UID: "d5f24e9f-9009-420a-80b8-1f7e5a4de630"). InnerVolumeSpecName "kube-api-access-gh8mc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:06.456325 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.456300    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh" (OuterVolumeSpecName: "kube-api-access-z24xh") pod "918732ca-85ce-47d8-ab21-57bd06cdaf46" (UID: "918732ca-85ce-47d8-ab21-57bd06cdaf46"). InnerVolumeSpecName "kube-api-access-z24xh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:06.555252 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.555170    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/918732ca-85ce-47d8-ab21-57bd06cdaf46-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:06.555252 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.555198    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z24xh\" (UniqueName: \"kubernetes.io/projected/918732ca-85ce-47d8-ab21-57bd06cdaf46-kube-api-access-z24xh\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:06.555252 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.555207    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gh8mc\" (UniqueName: \"kubernetes.io/projected/d5f24e9f-9009-420a-80b8-1f7e5a4de630-kube-api-access-gh8mc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:06.555252 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.555215    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5f24e9f-9009-420a-80b8-1f7e5a4de630-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:06.677280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.677253    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:37:06.681016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.680992    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-autores-a61c70-65cfd48689-vrwrq"]
Sep 11 13:37:06.691975 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.691947    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:37:06.696307 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.696283    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-autores-a61c70-87857849d-4xp6m"]
Sep 11 13:37:06.964423 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.964347    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4377c82e-df61-439c-b5ef-4ccb81f3ce9a" path="/var/lib/kubelet/pods/4377c82e-df61-439c-b5ef-4ccb81f3ce9a/volumes"
Sep 11 13:37:06.964763 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.964748    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="918732ca-85ce-47d8-ab21-57bd06cdaf46" path="/var/lib/kubelet/pods/918732ca-85ce-47d8-ab21-57bd06cdaf46/volumes"
Sep 11 13:37:06.965108 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:06.965095    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" path="/var/lib/kubelet/pods/d5f24e9f-9009-420a-80b8-1f7e5a4de630/volumes"
Sep 11 13:37:12.335005 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.334963    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:37:12.335517 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.335448    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" podUID="9afec799-e851-4930-b9b8-927382ec7fcf" containerName="payload-processing" containerID="cri-o://2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b" gracePeriod=30
Sep 11 13:37:12.341978 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.341952    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:37:12.342175 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.342153    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" podUID="34917548-bf6d-43a8-a846-68b04dc9bcb4" containerName="payload-pre-processing" containerID="cri-o://78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c" gracePeriod=30
Sep 11 13:37:12.613857 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.613829    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:37:12.616509 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.616491    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:37:12.646049 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.646023    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"]
Sep 11 13:37:12.655130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.655103    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-db87970d-lrh99"]
Sep 11 13:37:12.714722 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.714693    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume\") pod \"34917548-bf6d-43a8-a846-68b04dc9bcb4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") "
Sep 11 13:37:12.714722 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.714725    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume\") pod \"9afec799-e851-4930-b9b8-927382ec7fcf\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") "
Sep 11 13:37:12.714958 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.714759    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-wt7pc\" (UniqueName: \"kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc\") pod \"34917548-bf6d-43a8-a846-68b04dc9bcb4\" (UID: \"34917548-bf6d-43a8-a846-68b04dc9bcb4\") "
Sep 11 13:37:12.714958 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.714787    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2qdq\" (UniqueName: \"kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq\") pod \"9afec799-e851-4930-b9b8-927382ec7fcf\" (UID: \"9afec799-e851-4930-b9b8-927382ec7fcf\") "
Sep 11 13:37:12.715155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.715127    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "34917548-bf6d-43a8-a846-68b04dc9bcb4" (UID: "34917548-bf6d-43a8-a846-68b04dc9bcb4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:12.715254 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.715229    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "9afec799-e851-4930-b9b8-927382ec7fcf" (UID: "9afec799-e851-4930-b9b8-927382ec7fcf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:12.716968 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.716939    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq" (OuterVolumeSpecName: "kube-api-access-c2qdq") pod "9afec799-e851-4930-b9b8-927382ec7fcf" (UID: "9afec799-e851-4930-b9b8-927382ec7fcf"). InnerVolumeSpecName "kube-api-access-c2qdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:12.717064 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.716980    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc" (OuterVolumeSpecName: "kube-api-access-wt7pc") pod "34917548-bf6d-43a8-a846-68b04dc9bcb4" (UID: "34917548-bf6d-43a8-a846-68b04dc9bcb4"). InnerVolumeSpecName "kube-api-access-wt7pc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:12.816337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.816309    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/34917548-bf6d-43a8-a846-68b04dc9bcb4-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:12.816337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.816336    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/9afec799-e851-4930-b9b8-927382ec7fcf-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:12.816337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.816345    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wt7pc\" (UniqueName: \"kubernetes.io/projected/34917548-bf6d-43a8-a846-68b04dc9bcb4-kube-api-access-wt7pc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:12.816559 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.816353    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2qdq\" (UniqueName: \"kubernetes.io/projected/9afec799-e851-4930-b9b8-927382ec7fcf-kube-api-access-c2qdq\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:12.964164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:12.964091    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2458320a-d2c9-422e-a45f-c915c69a030e" path="/var/lib/kubelet/pods/2458320a-d2c9-422e-a45f-c915c69a030e/volumes"
Sep 11 13:37:13.393296 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.393268    2551 generic.go:356] "Generic (PLEG): container finished" podID="34917548-bf6d-43a8-a846-68b04dc9bcb4" containerID="78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c" exitCode=0
Sep 11 13:37:13.393752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.393314    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"
Sep 11 13:37:13.393752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.393364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" event={"ID":"34917548-bf6d-43a8-a846-68b04dc9bcb4","Type":"ContainerDied","Data":"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"}
Sep 11 13:37:13.393752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.393433    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4" event={"ID":"34917548-bf6d-43a8-a846-68b04dc9bcb4","Type":"ContainerDied","Data":"9777750d87d5a6449c5b5cfa2b5d1b2d9ae046873bdcaf48b650ce6fdaa9edcb"}
Sep 11 13:37:13.393752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.393469    2551 scope.go:117] "RemoveContainer" containerID="78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"
Sep 11 13:37:13.394873 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.394848    2551 generic.go:356] "Generic (PLEG): container finished" podID="9afec799-e851-4930-b9b8-927382ec7fcf" containerID="2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b" exitCode=0
Sep 11 13:37:13.395003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.394888    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"
Sep 11 13:37:13.395003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.394936    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" event={"ID":"9afec799-e851-4930-b9b8-927382ec7fcf","Type":"ContainerDied","Data":"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"}
Sep 11 13:37:13.395003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.394970    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc" event={"ID":"9afec799-e851-4930-b9b8-927382ec7fcf","Type":"ContainerDied","Data":"14eaa5c3c4b169a43b58587ee4b830ec3fa8c3bcd48e74760cde5e7c498a548c"}
Sep 11 13:37:13.402040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.402022    2551 scope.go:117] "RemoveContainer" containerID="78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"
Sep 11 13:37:13.402316 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:37:13.402296    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c\": container with ID starting with 78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c not found: ID does not exist" containerID="78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"
Sep 11 13:37:13.402402 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.402322    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c"} err="failed to get container status \"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c\": rpc error: code = NotFound desc = could not find container \"78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c\": container with ID starting with 78ff3c027ebbfaab0b4bebd6cd32e5c0c690fe8a1508a31a7b9fdcdbb98f234c not found: ID does not exist"
Sep 11 13:37:13.402402 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.402339    2551 scope.go:117] "RemoveContainer" containerID="2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"
Sep 11 13:37:13.409995 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.409978    2551 scope.go:117] "RemoveContainer" containerID="2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"
Sep 11 13:37:13.410254 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:37:13.410237    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b\": container with ID starting with 2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b not found: ID does not exist" containerID="2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"
Sep 11 13:37:13.410301 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.410261    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b"} err="failed to get container status \"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b\": rpc error: code = NotFound desc = could not find container \"2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b\": container with ID starting with 2959d9fa8badf4ed531aadc5fef4c79e9d48cb4e5c84c74c66e34d84a1499f7b not found: ID does not exist"
Sep 11 13:37:13.427683 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.427658    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:37:13.431377 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.431356    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-db87970d-748fc558d6-99jg4"]
Sep 11 13:37:13.446939 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.446915    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:37:13.454014 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:13.453990    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-db87970d-54d5696d77-r5brc"]
Sep 11 13:37:14.964238 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:14.964204    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="34917548-bf6d-43a8-a846-68b04dc9bcb4" path="/var/lib/kubelet/pods/34917548-bf6d-43a8-a846-68b04dc9bcb4/volumes"
Sep 11 13:37:14.964594 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:14.964559    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="9afec799-e851-4930-b9b8-927382ec7fcf" path="/var/lib/kubelet/pods/9afec799-e851-4930-b9b8-927382ec7fcf/volumes"
Sep 11 13:37:21.915547 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915472    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915818    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="918732ca-85ce-47d8-ab21-57bd06cdaf46" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915830    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="918732ca-85ce-47d8-ab21-57bd06cdaf46" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915841    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="9afec799-e851-4930-b9b8-927382ec7fcf" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915847    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="9afec799-e851-4930-b9b8-927382ec7fcf" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915856    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915861    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915867    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="34917548-bf6d-43a8-a846-68b04dc9bcb4" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915872    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="34917548-bf6d-43a8-a846-68b04dc9bcb4" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915882    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2458320a-d2c9-422e-a45f-c915c69a030e" containerName="revoke-keys"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915887    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2458320a-d2c9-422e-a45f-c915c69a030e" containerName="revoke-keys"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915961    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5f24e9f-9009-420a-80b8-1f7e5a4de630" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915973    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="918732ca-85ce-47d8-ab21-57bd06cdaf46" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915984    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="34917548-bf6d-43a8-a846-68b04dc9bcb4" containerName="payload-pre-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.915994    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="9afec799-e851-4930-b9b8-927382ec7fcf" containerName="payload-processing"
Sep 11 13:37:21.915988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.916004    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2458320a-d2c9-422e-a45f-c915c69a030e" containerName="revoke-keys"
Sep 11 13:37:21.925152 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.925117    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:21.928358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.928331    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-derive-344409b9\""
Sep 11 13:37:21.928751 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.928732    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-derive-344409b9-dockercfg-99p7l\""
Sep 11 13:37:21.934752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:21.934730    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:22.019059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.019026    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:22.022372 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.022356    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.039445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.039420    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:22.095040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.094999    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.095212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.095049    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.095212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.095164    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5r4mp\" (UniqueName: \"kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.095305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.095213    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xs97s\" (UniqueName: \"kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.196042 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.195959    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xs97s\" (UniqueName: \"kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.196042 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.196030    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.196273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.196054    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.196273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.196089    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5r4mp\" (UniqueName: \"kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.196838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.196808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.196944 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.196853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.208062 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.208032    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-xs97s\" (UniqueName: \"kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s\") pod \"payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") " pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.208387 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.208370    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-5r4mp\" (UniqueName: \"kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp\") pod \"payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") " pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.240363 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.240329    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:22.333133 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.333099    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:22.526996 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.526825    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:22.529682 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:37:22.529651    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0fc5b2b4_e59c_47ff_819b_d87c93e97039.slice/crio-73ea865c34d71ce7bea73d20c3e06ff1f8ac1ce28417cc2f447f5b51eac5aae6 WatchSource:0}: Error finding container 73ea865c34d71ce7bea73d20c3e06ff1f8ac1ce28417cc2f447f5b51eac5aae6: Status 404 returned error can't find the container with id 73ea865c34d71ce7bea73d20c3e06ff1f8ac1ce28417cc2f447f5b51eac5aae6
Sep 11 13:37:22.621836 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:22.621729    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:22.624544 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:37:22.624514    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30c2de38_4e6b_4e91_8b9f_d20539152652.slice/crio-173f92c19cea66ca18f23a1ace8c67b609b1eb7c882f96f0fc85809d3924f508 WatchSource:0}: Error finding container 173f92c19cea66ca18f23a1ace8c67b609b1eb7c882f96f0fc85809d3924f508: Status 404 returned error can't find the container with id 173f92c19cea66ca18f23a1ace8c67b609b1eb7c882f96f0fc85809d3924f508
Sep 11 13:37:23.446172 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.446137    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" event={"ID":"30c2de38-4e6b-4e91-8b9f-d20539152652","Type":"ContainerStarted","Data":"b328d1082511ff977b8bcc1fa6ef62c16b2b4422e8e03606d393c29e42c9770c"}
Sep 11 13:37:23.446778 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.446755    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" event={"ID":"30c2de38-4e6b-4e91-8b9f-d20539152652","Type":"ContainerStarted","Data":"173f92c19cea66ca18f23a1ace8c67b609b1eb7c882f96f0fc85809d3924f508"}
Sep 11 13:37:23.446924 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.446912    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:23.449122 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.449095    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" event={"ID":"0fc5b2b4-e59c-47ff-819b-d87c93e97039","Type":"ContainerStarted","Data":"17525fed560d66873fa7bb65bbc7dfb3d283fde4e68537bb8f063594b848f5eb"}
Sep 11 13:37:23.449254 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.449147    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:23.449254 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.449165    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" event={"ID":"0fc5b2b4-e59c-47ff-819b-d87c93e97039","Type":"ContainerStarted","Data":"73ea865c34d71ce7bea73d20c3e06ff1f8ac1ce28417cc2f447f5b51eac5aae6"}
Sep 11 13:37:23.471858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.471810    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" podStartSLOduration=2.471794273 podStartE2EDuration="2.471794273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:37:23.466512351 +0000 UTC m=+1074.967416799" watchObservedRunningTime="2026-09-11 13:37:23.471794273 +0000 UTC m=+1074.972698718"
Sep 11 13:37:23.500469 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:23.500406    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" podStartSLOduration=2.500383484 podStartE2EDuration="2.500383484s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:37:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:37:23.498551086 +0000 UTC m=+1074.999455539" watchObservedRunningTime="2026-09-11 13:37:23.500383484 +0000 UTC m=+1075.001287935"
Sep 11 13:37:31.850291 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.850259    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"]
Sep 11 13:37:31.855453 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.855426    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:31.857886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.857865    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:37:31.858104 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.858089    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:37:31.869140 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.866014    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"]
Sep 11 13:37:31.982248 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:31.982215    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-344409b9-d259g\" (UID: \"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:32.083124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:32.083087    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-344409b9-d259g\" (UID: \"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:32.083874 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:32.083853    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-derive-344409b9-d259g\" (UID: \"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:32.174893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:32.174805    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:32.447882 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:32.447790    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"]
Sep 11 13:37:32.453136 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:37:32.453090    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2ff1f77_cd2d_4f31_a7eb_99537a5715bf.slice/crio-9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee WatchSource:0}: Error finding container 9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee: Status 404 returned error can't find the container with id 9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee
Sep 11 13:37:32.496940 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:32.496913    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g" event={"ID":"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf","Type":"ContainerStarted","Data":"9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee"}
Sep 11 13:37:33.503683 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:33.503648    2551 generic.go:356] "Generic (PLEG): container finished" podID="e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" containerID="40755a4ac7ac25089826ef44995af808272c1074d4427ca0c1175bbcc11ca4da" exitCode=0
Sep 11 13:37:33.504083 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:33.503733    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g" event={"ID":"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf","Type":"ContainerDied","Data":"40755a4ac7ac25089826ef44995af808272c1074d4427ca0c1175bbcc11ca4da"}
Sep 11 13:37:34.457473 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:34.457444    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:34.647425 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:34.647395    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:34.807678 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:34.807648    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca\") pod \"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf\" (UID: \"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf\") "
Sep 11 13:37:34.807981 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:34.807958    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" (UID: "e2ff1f77-cd2d-4f31-a7eb-99537a5715bf"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:34.908244 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:34.908207    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:35.463249 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:35.463222    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:35.516038 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:35.516012    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"
Sep 11 13:37:35.516203 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:35.516055    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g" event={"ID":"e2ff1f77-cd2d-4f31-a7eb-99537a5715bf","Type":"ContainerDied","Data":"9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee"}
Sep 11 13:37:35.516203 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:35.516114    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9b96dc009300beecd962a433c0efe485646b9a24659eaa12cbfe729e4df19bee"
Sep 11 13:37:43.343010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.342973    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:43.343469 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.343225    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" podUID="30c2de38-4e6b-4e91-8b9f-d20539152652" containerName="payload-processing" containerID="cri-o://b328d1082511ff977b8bcc1fa6ef62c16b2b4422e8e03606d393c29e42c9770c" gracePeriod=30
Sep 11 13:37:43.356736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.356707    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:43.356998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.356966    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" podUID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" containerName="payload-pre-processing" containerID="cri-o://17525fed560d66873fa7bb65bbc7dfb3d283fde4e68537bb8f063594b848f5eb" gracePeriod=30
Sep 11 13:37:43.566458 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.566232    2551 generic.go:356] "Generic (PLEG): container finished" podID="30c2de38-4e6b-4e91-8b9f-d20539152652" containerID="b328d1082511ff977b8bcc1fa6ef62c16b2b4422e8e03606d393c29e42c9770c" exitCode=0
Sep 11 13:37:43.566458 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.566343    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" event={"ID":"30c2de38-4e6b-4e91-8b9f-d20539152652","Type":"ContainerDied","Data":"b328d1082511ff977b8bcc1fa6ef62c16b2b4422e8e03606d393c29e42c9770c"}
Sep 11 13:37:43.568295 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.568270    2551 generic.go:356] "Generic (PLEG): container finished" podID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" containerID="17525fed560d66873fa7bb65bbc7dfb3d283fde4e68537bb8f063594b848f5eb" exitCode=0
Sep 11 13:37:43.568423 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.568351    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" event={"ID":"0fc5b2b4-e59c-47ff-819b-d87c93e97039","Type":"ContainerDied","Data":"17525fed560d66873fa7bb65bbc7dfb3d283fde4e68537bb8f063594b848f5eb"}
Sep 11 13:37:43.625892 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.625868    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"]
Sep 11 13:37:43.630097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.629507    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-344409b9-d259g"]
Sep 11 13:37:43.642961 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.642939    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:43.646705 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.645992    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:43.685655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.685541    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume\") pod \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") "
Sep 11 13:37:43.686114 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686086    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume\") pod \"30c2de38-4e6b-4e91-8b9f-d20539152652\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") "
Sep 11 13:37:43.686294 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686276    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-5r4mp\" (UniqueName: \"kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp\") pod \"30c2de38-4e6b-4e91-8b9f-d20539152652\" (UID: \"30c2de38-4e6b-4e91-8b9f-d20539152652\") "
Sep 11 13:37:43.686489 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686465    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-xs97s\" (UniqueName: \"kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s\") pod \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\" (UID: \"0fc5b2b4-e59c-47ff-819b-d87c93e97039\") "
Sep 11 13:37:43.686603 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686506    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "30c2de38-4e6b-4e91-8b9f-d20539152652" (UID: "30c2de38-4e6b-4e91-8b9f-d20539152652"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:43.686774 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686753    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/30c2de38-4e6b-4e91-8b9f-d20539152652-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:43.687004 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.686976    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "0fc5b2b4-e59c-47ff-819b-d87c93e97039" (UID: "0fc5b2b4-e59c-47ff-819b-d87c93e97039"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:37:43.689439 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.689399    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp" (OuterVolumeSpecName: "kube-api-access-5r4mp") pod "30c2de38-4e6b-4e91-8b9f-d20539152652" (UID: "30c2de38-4e6b-4e91-8b9f-d20539152652"). InnerVolumeSpecName "kube-api-access-5r4mp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:43.689694 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.689664    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s" (OuterVolumeSpecName: "kube-api-access-xs97s") pod "0fc5b2b4-e59c-47ff-819b-d87c93e97039" (UID: "0fc5b2b4-e59c-47ff-819b-d87c93e97039"). InnerVolumeSpecName "kube-api-access-xs97s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:37:43.787603 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.787572    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5r4mp\" (UniqueName: \"kubernetes.io/projected/30c2de38-4e6b-4e91-8b9f-d20539152652-kube-api-access-5r4mp\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:43.787603 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.787600    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xs97s\" (UniqueName: \"kubernetes.io/projected/0fc5b2b4-e59c-47ff-819b-d87c93e97039-kube-api-access-xs97s\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:43.787603 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:43.787610    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/0fc5b2b4-e59c-47ff-819b-d87c93e97039-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:37:44.575546 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.575517    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"
Sep 11 13:37:44.576011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.575557    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q" event={"ID":"30c2de38-4e6b-4e91-8b9f-d20539152652","Type":"ContainerDied","Data":"173f92c19cea66ca18f23a1ace8c67b609b1eb7c882f96f0fc85809d3924f508"}
Sep 11 13:37:44.576011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.575619    2551 scope.go:117] "RemoveContainer" containerID="b328d1082511ff977b8bcc1fa6ef62c16b2b4422e8e03606d393c29e42c9770c"
Sep 11 13:37:44.577016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.576904    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5" event={"ID":"0fc5b2b4-e59c-47ff-819b-d87c93e97039","Type":"ContainerDied","Data":"73ea865c34d71ce7bea73d20c3e06ff1f8ac1ce28417cc2f447f5b51eac5aae6"}
Sep 11 13:37:44.577016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.576938    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"
Sep 11 13:37:44.584710 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.584542    2551 scope.go:117] "RemoveContainer" containerID="17525fed560d66873fa7bb65bbc7dfb3d283fde4e68537bb8f063594b848f5eb"
Sep 11 13:37:44.607472 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.607448    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:44.612472 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.612453    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-derive-344409b9-777549c97d-l95z5"]
Sep 11 13:37:44.627425 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.627371    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:44.632922 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.632899    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-derive-344409b9-5db49768b5-n8b9q"]
Sep 11 13:37:44.964858 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.964769    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" path="/var/lib/kubelet/pods/0fc5b2b4-e59c-47ff-819b-d87c93e97039/volumes"
Sep 11 13:37:44.965249 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.965228    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="30c2de38-4e6b-4e91-8b9f-d20539152652" path="/var/lib/kubelet/pods/30c2de38-4e6b-4e91-8b9f-d20539152652/volumes"
Sep 11 13:37:44.965568 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:44.965552    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" path="/var/lib/kubelet/pods/e2ff1f77-cd2d-4f31-a7eb-99537a5715bf/volumes"
Sep 11 13:37:55.419585 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419553    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:37:55.419942 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419920    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" containerName="payload-pre-processing"
Sep 11 13:37:55.419942 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419932    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" containerName="payload-pre-processing"
Sep 11 13:37:55.420011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419956    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" containerName="revoke-keys"
Sep 11 13:37:55.420011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419963    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" containerName="revoke-keys"
Sep 11 13:37:55.420011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419971    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="30c2de38-4e6b-4e91-8b9f-d20539152652" containerName="payload-processing"
Sep 11 13:37:55.420011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.419978    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="30c2de38-4e6b-4e91-8b9f-d20539152652" containerName="payload-processing"
Sep 11 13:37:55.420126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.420026    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e2ff1f77-cd2d-4f31-a7eb-99537a5715bf" containerName="revoke-keys"
Sep 11 13:37:55.420126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.420033    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="30c2de38-4e6b-4e91-8b9f-d20539152652" containerName="payload-processing"
Sep 11 13:37:55.420126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.420041    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0fc5b2b4-e59c-47ff-819b-d87c93e97039" containerName="payload-pre-processing"
Sep 11 13:37:55.424348 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.424331    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.427002 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.426979    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bfc5af\"/\"test-model-bfc5af-kserve-self-signed-certs\""
Sep 11 13:37:55.427754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.427729    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bfc5af\"/\"kube-root-ca.crt\""
Sep 11 13:37:55.427860 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.427756    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-inf-b-bfc5af\"/\"openshift-service-ca.crt\""
Sep 11 13:37:55.427930 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.427862    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-inf-b-bfc5af\"/\"default-dockercfg-cdgvj\""
Sep 11 13:37:55.433091 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.433067    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:37:55.489264 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489231    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsrxl\" (UniqueName: \"kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.489264 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489269    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.489511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489290    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.489511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489361    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.489511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489448    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.489511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.489465    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590593 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590555    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590593 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590591    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590846 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590654    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsrxl\" (UniqueName: \"kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590846 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590678    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590846 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590699    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.590846 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.590720    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.591047 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.591008    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.591093 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.591046    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.591093 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.591065    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.592940 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.592904    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.593207 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.593191    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.601317 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.601295    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsrxl\" (UniqueName: \"kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl\") pod \"test-model-bfc5af-kserve-545f78d575-fs8c4\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") " pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:55.734520 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:55.734471    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:37:56.010338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:56.010312    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:37:56.012131 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:37:56.012107    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod1eb6142d_0ec8_44c1_a7ab_195bd79f7c35.slice/crio-2766627b53767aee3789132a51eafa184d6fbd6ba892e3ac3918e086d83ff9e7 WatchSource:0}: Error finding container 2766627b53767aee3789132a51eafa184d6fbd6ba892e3ac3918e086d83ff9e7: Status 404 returned error can't find the container with id 2766627b53767aee3789132a51eafa184d6fbd6ba892e3ac3918e086d83ff9e7
Sep 11 13:37:56.638397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:56.638364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" event={"ID":"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35","Type":"ContainerStarted","Data":"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"}
Sep 11 13:37:56.638792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:56.638406    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" event={"ID":"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35","Type":"ContainerStarted","Data":"2766627b53767aee3789132a51eafa184d6fbd6ba892e3ac3918e086d83ff9e7"}
Sep 11 13:37:56.658399 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:37:56.658346    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" podStartSLOduration=1.658330246 podStartE2EDuration="1.658330246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:37:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:37:56.656469735 +0000 UTC m=+1108.157374197" watchObservedRunningTime="2026-09-11 13:37:56.658330246 +0000 UTC m=+1108.159234739"
Sep 11 13:38:05.735462 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:05.735423    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:38:05.735962 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:05.735475    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:38:05.748549 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:05.748520    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:38:06.699750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:06.699720    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:38:38.991324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:38.991292    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"]
Sep 11 13:38:38.996040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:38.996022    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:38.998508 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:38.998490    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:38:38.998603 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:38.998525    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:38:39.003759 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.003737    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"]
Sep 11 13:38:39.060147 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.060113    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4\" (UID: \"f1276836-6a5c-47e7-bb97-43e9cf34f4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:39.161592 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.161553    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4\" (UID: \"f1276836-6a5c-47e7-bb97-43e9cf34f4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:39.162257 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.162230    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4\" (UID: \"f1276836-6a5c-47e7-bb97-43e9cf34f4d1\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:39.307712 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.307596    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:39.590974 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.590947    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"]
Sep 11 13:38:39.592604 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:38:39.592577    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf1276836_6a5c_47e7_bb97_43e9cf34f4d1.slice/crio-5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f WatchSource:0}: Error finding container 5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f: Status 404 returned error can't find the container with id 5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f
Sep 11 13:38:39.848960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:39.848923    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4" event={"ID":"f1276836-6a5c-47e7-bb97-43e9cf34f4d1","Type":"ContainerStarted","Data":"5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f"}
Sep 11 13:38:40.854960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:40.854930    2551 generic.go:356] "Generic (PLEG): container finished" podID="f1276836-6a5c-47e7-bb97-43e9cf34f4d1" containerID="485108e1c0e4360de7ff674066b191d347e6104025a26b10b70a13437474bf3f" exitCode=0
Sep 11 13:38:40.855351 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:40.855015    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4" event={"ID":"f1276836-6a5c-47e7-bb97-43e9cf34f4d1","Type":"ContainerDied","Data":"485108e1c0e4360de7ff674066b191d347e6104025a26b10b70a13437474bf3f"}
Sep 11 13:38:41.993178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:41.993155    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:42.086795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.086764    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca\") pod \"f1276836-6a5c-47e7-bb97-43e9cf34f4d1\" (UID: \"f1276836-6a5c-47e7-bb97-43e9cf34f4d1\") "
Sep 11 13:38:42.087127 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.087101    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "f1276836-6a5c-47e7-bb97-43e9cf34f4d1" (UID: "f1276836-6a5c-47e7-bb97-43e9cf34f4d1"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:38:42.187400 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.187311    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f1276836-6a5c-47e7-bb97-43e9cf34f4d1-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:38:42.866402 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.866374    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"
Sep 11 13:38:42.866573 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.866364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4" event={"ID":"f1276836-6a5c-47e7-bb97-43e9cf34f4d1","Type":"ContainerDied","Data":"5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f"}
Sep 11 13:38:42.866573 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:42.866541    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5cb10091ee289c5553611d767045f703526669d43a19c004c274d31bc5f6669f"
Sep 11 13:38:50.059015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:50.058979    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"]
Sep 11 13:38:50.061289 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:50.061264    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-explicit-cdac79-g2lq4"]
Sep 11 13:38:50.964464 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:38:50.964431    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="f1276836-6a5c-47e7-bb97-43e9cf34f4d1" path="/var/lib/kubelet/pods/f1276836-6a5c-47e7-bb97-43e9cf34f4d1/volumes"
Sep 11 13:39:31.479209 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:31.479170    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:39:32.467100 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.467066    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:39:32.598963 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.598932    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"]
Sep 11 13:39:32.599333 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.599271    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="f1276836-6a5c-47e7-bb97-43e9cf34f4d1" containerName="revoke-keys"
Sep 11 13:39:32.599333 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.599282    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="f1276836-6a5c-47e7-bb97-43e9cf34f4d1" containerName="revoke-keys"
Sep 11 13:39:32.599404 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.599351    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="f1276836-6a5c-47e7-bb97-43e9cf34f4d1" containerName="revoke-keys"
Sep 11 13:39:32.602321 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.602304    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:32.604967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.604943    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:39:32.605177 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.605156    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:39:32.615471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.615446    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"]
Sep 11 13:39:32.722484 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.722457    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-94126e-p88dz\" (UID: \"cd8fabe4-f0af-42e5-8478-e927ce642da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:32.823557 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.823526    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-94126e-p88dz\" (UID: \"cd8fabe4-f0af-42e5-8478-e927ce642da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:32.824155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.824122    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-94126e-p88dz\" (UID: \"cd8fabe4-f0af-42e5-8478-e927ce642da4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:32.912034 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:32.912002    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:33.190098 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:33.190076    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"]
Sep 11 13:39:33.192877 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:39:33.192841    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcd8fabe4_f0af_42e5_8478_e927ce642da4.slice/crio-553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c WatchSource:0}: Error finding container 553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c: Status 404 returned error can't find the container with id 553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c
Sep 11 13:39:34.130705 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:34.130601    2551 generic.go:356] "Generic (PLEG): container finished" podID="cd8fabe4-f0af-42e5-8478-e927ce642da4" containerID="077e5bf5fd68ff8f17fa161f6380ec2d1159dfe817724c0fefa580ecc592ed98" exitCode=0
Sep 11 13:39:34.130705 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:34.130682    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz" event={"ID":"cd8fabe4-f0af-42e5-8478-e927ce642da4","Type":"ContainerDied","Data":"077e5bf5fd68ff8f17fa161f6380ec2d1159dfe817724c0fefa580ecc592ed98"}
Sep 11 13:39:34.131133 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:34.130737    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz" event={"ID":"cd8fabe4-f0af-42e5-8478-e927ce642da4","Type":"ContainerStarted","Data":"553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c"}
Sep 11 13:39:35.276022 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:35.275998    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:35.347623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:35.347581    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca\") pod \"cd8fabe4-f0af-42e5-8478-e927ce642da4\" (UID: \"cd8fabe4-f0af-42e5-8478-e927ce642da4\") "
Sep 11 13:39:35.347937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:35.347912    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cd8fabe4-f0af-42e5-8478-e927ce642da4" (UID: "cd8fabe4-f0af-42e5-8478-e927ce642da4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:39:35.448920 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:35.448840    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cd8fabe4-f0af-42e5-8478-e927ce642da4-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:36.140956 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:36.140921    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"
Sep 11 13:39:36.141139 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:36.140952    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz" event={"ID":"cd8fabe4-f0af-42e5-8478-e927ce642da4","Type":"ContainerDied","Data":"553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c"}
Sep 11 13:39:36.141139 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:36.141004    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="553ff606eb3ede47165cc1e82792c5ce5c9c68d66394b88f62343324a85cff9c"
Sep 11 13:39:43.449371 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.449338    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:39:43.449981 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.449594    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" podUID="e35498b9-ace4-40b5-8580-6c6a7caff743" containerName="payload-processing" containerID="cri-o://10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e" gracePeriod=30
Sep 11 13:39:43.453857 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.453825    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:39:43.454132 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.454102    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" podUID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" containerName="payload-pre-processing" containerID="cri-o://e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b" gracePeriod=30
Sep 11 13:39:43.700599 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.700532    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:39:43.731668 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.731613    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:39:43.750370 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.750340    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"]
Sep 11 13:39:43.753150 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.753118    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-94126e-p88dz"]
Sep 11 13:39:43.828220 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828187    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qwtgt\" (UniqueName: \"kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt\") pod \"e35498b9-ace4-40b5-8580-6c6a7caff743\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") "
Sep 11 13:39:43.828220 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828222    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume\") pod \"e35498b9-ace4-40b5-8580-6c6a7caff743\" (UID: \"e35498b9-ace4-40b5-8580-6c6a7caff743\") "
Sep 11 13:39:43.828450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828282    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-dvtmc\" (UniqueName: \"kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc\") pod \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") "
Sep 11 13:39:43.828450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828334    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume\") pod \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\" (UID: \"edbe8644-c1d7-4e70-af3e-3a4adc85156b\") "
Sep 11 13:39:43.828736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828705    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "e35498b9-ace4-40b5-8580-6c6a7caff743" (UID: "e35498b9-ace4-40b5-8580-6c6a7caff743"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:39:43.828819 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.828724    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "edbe8644-c1d7-4e70-af3e-3a4adc85156b" (UID: "edbe8644-c1d7-4e70-af3e-3a4adc85156b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:39:43.830258 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.830234    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt" (OuterVolumeSpecName: "kube-api-access-qwtgt") pod "e35498b9-ace4-40b5-8580-6c6a7caff743" (UID: "e35498b9-ace4-40b5-8580-6c6a7caff743"). InnerVolumeSpecName "kube-api-access-qwtgt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:39:43.830309 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.830289    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc" (OuterVolumeSpecName: "kube-api-access-dvtmc") pod "edbe8644-c1d7-4e70-af3e-3a4adc85156b" (UID: "edbe8644-c1d7-4e70-af3e-3a4adc85156b"). InnerVolumeSpecName "kube-api-access-dvtmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:39:43.929176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.929143    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qwtgt\" (UniqueName: \"kubernetes.io/projected/e35498b9-ace4-40b5-8580-6c6a7caff743-kube-api-access-qwtgt\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:43.929176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.929172    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e35498b9-ace4-40b5-8580-6c6a7caff743-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:43.929176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.929183    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvtmc\" (UniqueName: \"kubernetes.io/projected/edbe8644-c1d7-4e70-af3e-3a4adc85156b-kube-api-access-dvtmc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:43.929429 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:43.929192    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/edbe8644-c1d7-4e70-af3e-3a4adc85156b-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:44.184684 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.184618    2551 generic.go:356] "Generic (PLEG): container finished" podID="e35498b9-ace4-40b5-8580-6c6a7caff743" containerID="10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e" exitCode=0
Sep 11 13:39:44.184684 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.184687    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"
Sep 11 13:39:44.184926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.184731    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" event={"ID":"e35498b9-ace4-40b5-8580-6c6a7caff743","Type":"ContainerDied","Data":"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"}
Sep 11 13:39:44.184926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.184780    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l" event={"ID":"e35498b9-ace4-40b5-8580-6c6a7caff743","Type":"ContainerDied","Data":"b963b3e1b2b091c29db9ecbefa00b7a0ff0eeb04654a4bf21987393a41237f37"}
Sep 11 13:39:44.184926 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.184804    2551 scope.go:117] "RemoveContainer" containerID="10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"
Sep 11 13:39:44.186235 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.186208    2551 generic.go:356] "Generic (PLEG): container finished" podID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" containerID="e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b" exitCode=0
Sep 11 13:39:44.186341 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.186257    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"
Sep 11 13:39:44.186341 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.186289    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" event={"ID":"edbe8644-c1d7-4e70-af3e-3a4adc85156b","Type":"ContainerDied","Data":"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"}
Sep 11 13:39:44.186341 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.186336    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt" event={"ID":"edbe8644-c1d7-4e70-af3e-3a4adc85156b","Type":"ContainerDied","Data":"8ecb06a49d9742eaced9cf9f90a3d860c5c3a1f62101629740a3439b5ecc2118"}
Sep 11 13:39:44.198216 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.198119    2551 scope.go:117] "RemoveContainer" containerID="10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"
Sep 11 13:39:44.198489 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:39:44.198460    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e\": container with ID starting with 10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e not found: ID does not exist" containerID="10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"
Sep 11 13:39:44.198577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.198502    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e"} err="failed to get container status \"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e\": rpc error: code = NotFound desc = could not find container \"10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e\": container with ID starting with 10fd24bddb49e765c82d4bb060a0506d6527db91b6682e1d7cad0c5c9798074e not found: ID does not exist"
Sep 11 13:39:44.198577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.198526    2551 scope.go:117] "RemoveContainer" containerID="e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"
Sep 11 13:39:44.206528 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.206503    2551 scope.go:117] "RemoveContainer" containerID="e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"
Sep 11 13:39:44.206836 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:39:44.206818    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b\": container with ID starting with e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b not found: ID does not exist" containerID="e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"
Sep 11 13:39:44.206916 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.206847    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b"} err="failed to get container status \"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b\": rpc error: code = NotFound desc = could not find container \"e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b\": container with ID starting with e1fe713302f1053bfd6c1001d78d9d6d46327001dc964409c46074c87412b15b not found: ID does not exist"
Sep 11 13:39:44.207253 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.207235    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:39:44.214261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.214237    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-94126e-7c6d5f555f-28n7l"]
Sep 11 13:39:44.224645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.224608    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:39:44.229620 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.229599    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-94126e-5bb77786b6-nt4mt"]
Sep 11 13:39:44.738244 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738206    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:39:44.738609 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738588    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" containerName="payload-pre-processing"
Sep 11 13:39:44.738609 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738602    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" containerName="payload-pre-processing"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738612    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="e35498b9-ace4-40b5-8580-6c6a7caff743" containerName="payload-processing"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738618    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="e35498b9-ace4-40b5-8580-6c6a7caff743" containerName="payload-processing"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738639    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cd8fabe4-f0af-42e5-8478-e927ce642da4" containerName="revoke-keys"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738646    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cd8fabe4-f0af-42e5-8478-e927ce642da4" containerName="revoke-keys"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738693    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="e35498b9-ace4-40b5-8580-6c6a7caff743" containerName="payload-processing"
Sep 11 13:39:44.738706 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738704    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cd8fabe4-f0af-42e5-8478-e927ce642da4" containerName="revoke-keys"
Sep 11 13:39:44.738897 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.738710    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" containerName="payload-pre-processing"
Sep 11 13:39:44.742542 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.742524    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.744962 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.744939    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-62b0dfd4\""
Sep 11 13:39:44.744962 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.744957    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-62b0dfd4-dockercfg-27jd4\""
Sep 11 13:39:44.750816 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.750325    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:39:44.788868 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.788837    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:39:44.792125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.792108    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:44.801570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.801544    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:39:44.838096 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.838065    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.838260 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.838110    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.939191 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.939142    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.939191 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.939202    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:44.939437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.939236    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sf9kx\" (UniqueName: \"kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:44.939437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.939305    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.939973 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.939954    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.947998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.947976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8\") pod \"payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:44.964001 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.963973    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cd8fabe4-f0af-42e5-8478-e927ce642da4" path="/var/lib/kubelet/pods/cd8fabe4-f0af-42e5-8478-e927ce642da4/volumes"
Sep 11 13:39:44.964417 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.964398    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="e35498b9-ace4-40b5-8580-6c6a7caff743" path="/var/lib/kubelet/pods/e35498b9-ace4-40b5-8580-6c6a7caff743/volumes"
Sep 11 13:39:44.964818 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:44.964801    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="edbe8644-c1d7-4e70-af3e-3a4adc85156b" path="/var/lib/kubelet/pods/edbe8644-c1d7-4e70-af3e-3a4adc85156b/volumes"
Sep 11 13:39:45.040602 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.040514    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:45.040602 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.040568    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sf9kx\" (UniqueName: \"kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:45.041332 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.041315    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:45.049995 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.049966    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sf9kx\" (UniqueName: \"kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx\") pod \"payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") " pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:45.054566 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.054545    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:45.103159 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.103124    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:45.327817 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.327793    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:39:45.329879 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:39:45.329852    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda61ca4da_c59e_4e68_a9f3_168dfa552568.slice/crio-d96acb85e7adf2728a6364ee80a8bb3906c14c159d6994fb9490cb623549ee07 WatchSource:0}: Error finding container d96acb85e7adf2728a6364ee80a8bb3906c14c159d6994fb9490cb623549ee07: Status 404 returned error can't find the container with id d96acb85e7adf2728a6364ee80a8bb3906c14c159d6994fb9490cb623549ee07
Sep 11 13:39:45.388968 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:45.388937    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:39:45.391769 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:39:45.391745    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb439611f_ba22_4bcc_8480_03db3fc9b38c.slice/crio-0b0a2cafaf424c13268829e1c62bad74b35d6960c2aec2824de7c24151e414e4 WatchSource:0}: Error finding container 0b0a2cafaf424c13268829e1c62bad74b35d6960c2aec2824de7c24151e414e4: Status 404 returned error can't find the container with id 0b0a2cafaf424c13268829e1c62bad74b35d6960c2aec2824de7c24151e414e4
Sep 11 13:39:46.201082 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.201049    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" event={"ID":"b439611f-ba22-4bcc-8480-03db3fc9b38c","Type":"ContainerStarted","Data":"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"}
Sep 11 13:39:46.201540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.201103    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" event={"ID":"b439611f-ba22-4bcc-8480-03db3fc9b38c","Type":"ContainerStarted","Data":"0b0a2cafaf424c13268829e1c62bad74b35d6960c2aec2824de7c24151e414e4"}
Sep 11 13:39:46.201540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.201166    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:39:46.202679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.202655    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" event={"ID":"a61ca4da-c59e-4e68-a9f3-168dfa552568","Type":"ContainerStarted","Data":"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"}
Sep 11 13:39:46.202806 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.202702    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" event={"ID":"a61ca4da-c59e-4e68-a9f3-168dfa552568","Type":"ContainerStarted","Data":"d96acb85e7adf2728a6364ee80a8bb3906c14c159d6994fb9490cb623549ee07"}
Sep 11 13:39:46.202806 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.202751    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:46.219966 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.219915    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" podStartSLOduration=2.219899352 podStartE2EDuration="2.219899352s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:39:46.218907447 +0000 UTC m=+1217.719811921" watchObservedRunningTime="2026-09-11 13:39:46.219899352 +0000 UTC m=+1217.720803801"
Sep 11 13:39:46.241127 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.241062    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" podStartSLOduration=2.241038316 podStartE2EDuration="2.241038316s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:39:46.236054376 +0000 UTC m=+1217.736958851" watchObservedRunningTime="2026-09-11 13:39:46.241038316 +0000 UTC m=+1217.741942766"
Sep 11 13:39:46.498225 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.498160    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" podUID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" containerName="main" containerID="cri-o://00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c" gracePeriod=45
Sep 11 13:39:46.748029 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.748004    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:39:46.858131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858037    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858106    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858139    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858165    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-4rfj5\" (UniqueName: \"kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858193    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858252    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home\") pod \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\" (UID: \"0f5d9c6e-9caf-4425-ad44-960b9d1326b6\") "
Sep 11 13:39:46.858576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858415    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache" (OuterVolumeSpecName: "model-cache") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:46.858576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858429    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:46.858576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858539    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:46.858576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858562    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:46.858768 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.858643    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home" (OuterVolumeSpecName: "home") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:46.860501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.860472    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:39:46.860501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.860485    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5" (OuterVolumeSpecName: "kube-api-access-4rfj5") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "kube-api-access-4rfj5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:39:46.860676 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.860547    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm" (OuterVolumeSpecName: "dshm") pod "0f5d9c6e-9caf-4425-ad44-960b9d1326b6" (UID: "0f5d9c6e-9caf-4425-ad44-960b9d1326b6"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:46.959346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.959240    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:46.959346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.959274    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:46.959346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.959288    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4rfj5\" (UniqueName: \"kubernetes.io/projected/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-kube-api-access-4rfj5\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:46.959346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:46.959303    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0f5d9c6e-9caf-4425-ad44-960b9d1326b6-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.208912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.208824    2551 generic.go:356] "Generic (PLEG): container finished" podID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" containerID="00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c" exitCode=0
Sep 11 13:39:47.208912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.208866    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"
Sep 11 13:39:47.209354 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.208913    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" event={"ID":"0f5d9c6e-9caf-4425-ad44-960b9d1326b6","Type":"ContainerDied","Data":"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"}
Sep 11 13:39:47.209354 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.208958    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v" event={"ID":"0f5d9c6e-9caf-4425-ad44-960b9d1326b6","Type":"ContainerDied","Data":"6fd6a79ef90f2afc951cef8db1ef4bcc500298ae6178702b4cacf8dad28a44d4"}
Sep 11 13:39:47.209354 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.208976    2551 scope.go:117] "RemoveContainer" containerID="00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"
Sep 11 13:39:47.217730 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.217711    2551 scope.go:117] "RemoveContainer" containerID="00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"
Sep 11 13:39:47.218010 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:39:47.217991    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c\": container with ID starting with 00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c not found: ID does not exist" containerID="00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"
Sep 11 13:39:47.218114 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.218019    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c"} err="failed to get container status \"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c\": rpc error: code = NotFound desc = could not find container \"00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c\": container with ID starting with 00520bfe83353ce6f27d9696a45c69d1b21e4865082ab66858137c1fbe19560c not found: ID does not exist"
Sep 11 13:39:47.273271 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.273235    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:39:47.278159 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.278130    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-a-94126e/test-model-94126e-kserve-785769d4d7-tsn6v"]
Sep 11 13:39:47.485759 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.485715    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" podUID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" containerName="main" containerID="cri-o://0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31" gracePeriod=45
Sep 11 13:39:47.742010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.741950    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:39:47.868324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868288    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868332    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868541 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868355    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868541 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868371    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868541 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868422    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-rsrxl\" (UniqueName: \"kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868541 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868443    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home\") pod \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\" (UID: \"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35\") "
Sep 11 13:39:47.868795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868549    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:47.868795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868673    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache" (OuterVolumeSpecName: "model-cache") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:47.868795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868762    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home" (OuterVolumeSpecName: "home") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:47.868957 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868809    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.868957 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.868822    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.870666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.870643    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm" (OuterVolumeSpecName: "dshm") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:39:47.870755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.870676    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:39:47.870755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.870684    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl" (OuterVolumeSpecName: "kube-api-access-rsrxl") pod "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" (UID: "1eb6142d-0ec8-44c1-a7ab-195bd79f7c35"). InnerVolumeSpecName "kube-api-access-rsrxl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:39:47.969935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.969898    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.969935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.969928    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.969935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.969941    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsrxl\" (UniqueName: \"kubernetes.io/projected/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-kube-api-access-rsrxl\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:47.970160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:47.969950    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:48.216034 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.215949    2551 generic.go:356] "Generic (PLEG): container finished" podID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" containerID="0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31" exitCode=0
Sep 11 13:39:48.216034 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.215991    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"
Sep 11 13:39:48.216544 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.216039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" event={"ID":"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35","Type":"ContainerDied","Data":"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"}
Sep 11 13:39:48.216544 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.216085    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4" event={"ID":"1eb6142d-0ec8-44c1-a7ab-195bd79f7c35","Type":"ContainerDied","Data":"2766627b53767aee3789132a51eafa184d6fbd6ba892e3ac3918e086d83ff9e7"}
Sep 11 13:39:48.216544 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.216108    2551 scope.go:117] "RemoveContainer" containerID="0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"
Sep 11 13:39:48.224828 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.224811    2551 scope.go:117] "RemoveContainer" containerID="0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"
Sep 11 13:39:48.225098 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:39:48.225080    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31\": container with ID starting with 0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31 not found: ID does not exist" containerID="0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"
Sep 11 13:39:48.225154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.225107    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31"} err="failed to get container status \"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31\": rpc error: code = NotFound desc = could not find container \"0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31\": container with ID starting with 0c18c28594a6ff69691b09940b67b667c74778ebd1ef83ecabe95c753152db31 not found: ID does not exist"
Sep 11 13:39:48.240242 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.240222    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:39:48.242780 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.242760    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-inf-b-bfc5af/test-model-bfc5af-kserve-545f78d575-fs8c4"]
Sep 11 13:39:48.963998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.963966    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" path="/var/lib/kubelet/pods/0f5d9c6e-9caf-4425-ad44-960b9d1326b6/volumes"
Sep 11 13:39:48.964350 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:48.964337    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" path="/var/lib/kubelet/pods/1eb6142d-0ec8-44c1-a7ab-195bd79f7c35/volumes"
Sep 11 13:39:50.768600 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.768556    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"]
Sep 11 13:39:50.769135 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769114    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" containerName="main"
Sep 11 13:39:50.769220 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769139    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" containerName="main"
Sep 11 13:39:50.769220 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769152    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" containerName="main"
Sep 11 13:39:50.769220 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769160    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" containerName="main"
Sep 11 13:39:50.769370 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769244    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="0f5d9c6e-9caf-4425-ad44-960b9d1326b6" containerName="main"
Sep 11 13:39:50.769370 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.769262    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="1eb6142d-0ec8-44c1-a7ab-195bd79f7c35" containerName="main"
Sep 11 13:39:50.773616 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.773595    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:50.776142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.776116    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:39:50.776267 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.776199    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:39:50.781503 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.781481    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"]
Sep 11 13:39:50.896262 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.896221    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2\" (UID: \"6255d6ef-3277-4547-8c8e-32cb7e1f7468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:50.997167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.997131    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2\" (UID: \"6255d6ef-3277-4547-8c8e-32cb7e1f7468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:50.997783 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:50.997766    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2\" (UID: \"6255d6ef-3277-4547-8c8e-32cb7e1f7468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:51.085000 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:51.084907    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:51.355923 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:51.355891    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"]
Sep 11 13:39:51.358535 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:39:51.358513    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6255d6ef_3277_4547_8c8e_32cb7e1f7468.slice/crio-fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f WatchSource:0}: Error finding container fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f: Status 404 returned error can't find the container with id fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f
Sep 11 13:39:52.238584 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:52.238552    2551 generic.go:356] "Generic (PLEG): container finished" podID="6255d6ef-3277-4547-8c8e-32cb7e1f7468" containerID="b7f6de540c82a61a63af9964b6f9663c79694bcb2dac50c2233d84b6941895c4" exitCode=0
Sep 11 13:39:52.238960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:52.238644    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2" event={"ID":"6255d6ef-3277-4547-8c8e-32cb7e1f7468","Type":"ContainerDied","Data":"b7f6de540c82a61a63af9964b6f9663c79694bcb2dac50c2233d84b6941895c4"}
Sep 11 13:39:52.238960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:52.238694    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2" event={"ID":"6255d6ef-3277-4547-8c8e-32cb7e1f7468","Type":"ContainerStarted","Data":"fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f"}
Sep 11 13:39:53.377092 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:53.377063    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:53.520369 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:53.520277    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca\") pod \"6255d6ef-3277-4547-8c8e-32cb7e1f7468\" (UID: \"6255d6ef-3277-4547-8c8e-32cb7e1f7468\") "
Sep 11 13:39:53.520661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:53.520621    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "6255d6ef-3277-4547-8c8e-32cb7e1f7468" (UID: "6255d6ef-3277-4547-8c8e-32cb7e1f7468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:39:53.621797 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:53.621757    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/6255d6ef-3277-4547-8c8e-32cb7e1f7468-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:39:54.250707 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:54.250670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2" event={"ID":"6255d6ef-3277-4547-8c8e-32cb7e1f7468","Type":"ContainerDied","Data":"fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f"}
Sep 11 13:39:54.250884 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:54.250725    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fb0a5e5d872c9cde41000427c3c7fddb6b705baab9e031422f8e37a2360d021f"
Sep 11 13:39:54.250884 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:54.250687    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"
Sep 11 13:39:57.211645 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:57.211595    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:39:57.212102 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:39:57.211709    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:40:01.610548 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.610511    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:40:01.610966 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.610775    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" podUID="871f39e3-4467-4172-8680-9469517b9fc9" containerName="payload-processing" containerID="cri-o://24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937" gracePeriod=30
Sep 11 13:40:01.620350 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.620324    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:40:01.621076 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.620559    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" podUID="96e88044-00ed-4ea2-9157-b9454711bf29" containerName="payload-pre-processing" containerID="cri-o://38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0" gracePeriod=30
Sep 11 13:40:01.852021 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.851973    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"]
Sep 11 13:40:01.857700 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.857676    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-bfc5af-gtwh2"]
Sep 11 13:40:01.872818 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.872758    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:40:01.893814 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.893790    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume\") pod \"871f39e3-4467-4172-8680-9469517b9fc9\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") "
Sep 11 13:40:01.893962 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.893874    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-szlh4\" (UniqueName: \"kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4\") pod \"871f39e3-4467-4172-8680-9469517b9fc9\" (UID: \"871f39e3-4467-4172-8680-9469517b9fc9\") "
Sep 11 13:40:01.894163 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.894139    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "871f39e3-4467-4172-8680-9469517b9fc9" (UID: "871f39e3-4467-4172-8680-9469517b9fc9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:01.896059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.896031    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4" (OuterVolumeSpecName: "kube-api-access-szlh4") pod "871f39e3-4467-4172-8680-9469517b9fc9" (UID: "871f39e3-4467-4172-8680-9469517b9fc9"). InnerVolumeSpecName "kube-api-access-szlh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:40:01.907381 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.907357    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:40:01.994461 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.994424    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-q6dbd\" (UniqueName: \"kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd\") pod \"96e88044-00ed-4ea2-9157-b9454711bf29\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") "
Sep 11 13:40:01.994461 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.994470    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume\") pod \"96e88044-00ed-4ea2-9157-b9454711bf29\" (UID: \"96e88044-00ed-4ea2-9157-b9454711bf29\") "
Sep 11 13:40:01.994779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.994736    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/871f39e3-4467-4172-8680-9469517b9fc9-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:01.994779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.994758    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-szlh4\" (UniqueName: \"kubernetes.io/projected/871f39e3-4467-4172-8680-9469517b9fc9-kube-api-access-szlh4\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:01.994915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.994891    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "96e88044-00ed-4ea2-9157-b9454711bf29" (UID: "96e88044-00ed-4ea2-9157-b9454711bf29"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:01.996563 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:01.996542    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd" (OuterVolumeSpecName: "kube-api-access-q6dbd") pod "96e88044-00ed-4ea2-9157-b9454711bf29" (UID: "96e88044-00ed-4ea2-9157-b9454711bf29"). InnerVolumeSpecName "kube-api-access-q6dbd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:40:02.096154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.096116    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6dbd\" (UniqueName: \"kubernetes.io/projected/96e88044-00ed-4ea2-9157-b9454711bf29-kube-api-access-q6dbd\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:02.096154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.096147    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96e88044-00ed-4ea2-9157-b9454711bf29-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:02.290433 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.290393    2551 generic.go:356] "Generic (PLEG): container finished" podID="871f39e3-4467-4172-8680-9469517b9fc9" containerID="24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937" exitCode=0
Sep 11 13:40:02.290611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.290457    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"
Sep 11 13:40:02.290611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.290475    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" event={"ID":"871f39e3-4467-4172-8680-9469517b9fc9","Type":"ContainerDied","Data":"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"}
Sep 11 13:40:02.290611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.290567    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb" event={"ID":"871f39e3-4467-4172-8680-9469517b9fc9","Type":"ContainerDied","Data":"66b485a17a2e3e0a46ea2013099def0cbc3882f1419bd0b006f499f76090b6cb"}
Sep 11 13:40:02.290611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.290595    2551 scope.go:117] "RemoveContainer" containerID="24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"
Sep 11 13:40:02.291934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.291907    2551 generic.go:356] "Generic (PLEG): container finished" podID="96e88044-00ed-4ea2-9157-b9454711bf29" containerID="38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0" exitCode=0
Sep 11 13:40:02.292051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.291959    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"
Sep 11 13:40:02.292051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.291963    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" event={"ID":"96e88044-00ed-4ea2-9157-b9454711bf29","Type":"ContainerDied","Data":"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"}
Sep 11 13:40:02.292051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.292012    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9" event={"ID":"96e88044-00ed-4ea2-9157-b9454711bf29","Type":"ContainerDied","Data":"e06049c40c9fe46937283afc6bd5f76bf1b8e4b2a6c46d59e1c06c895dea42db"}
Sep 11 13:40:02.299566 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.299550    2551 scope.go:117] "RemoveContainer" containerID="24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"
Sep 11 13:40:02.299889 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:40:02.299864    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937\": container with ID starting with 24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937 not found: ID does not exist" containerID="24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"
Sep 11 13:40:02.299964 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.299897    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937"} err="failed to get container status \"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937\": rpc error: code = NotFound desc = could not find container \"24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937\": container with ID starting with 24ed00d03050f185d7212d5dbcf5cc902b8cb08c85afabac48ec2862b809a937 not found: ID does not exist"
Sep 11 13:40:02.299964 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.299915    2551 scope.go:117] "RemoveContainer" containerID="38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"
Sep 11 13:40:02.308342 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.308323    2551 scope.go:117] "RemoveContainer" containerID="38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"
Sep 11 13:40:02.308652 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:40:02.308611    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0\": container with ID starting with 38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0 not found: ID does not exist" containerID="38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"
Sep 11 13:40:02.308737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.308655    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0"} err="failed to get container status \"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0\": rpc error: code = NotFound desc = could not find container \"38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0\": container with ID starting with 38ce485493a769754a79554cc54bc97288cf633d60f411f271dc0d06e1b2d0c0 not found: ID does not exist"
Sep 11 13:40:02.317149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.317122    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:40:02.322428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.322404    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-bfc5af-5c686fcc65-skxkb"]
Sep 11 13:40:02.339376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.339346    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:40:02.354350 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.354313    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-bfc5af-8dbb4dd4b-mbdw9"]
Sep 11 13:40:02.964231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.964152    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="6255d6ef-3277-4547-8c8e-32cb7e1f7468" path="/var/lib/kubelet/pods/6255d6ef-3277-4547-8c8e-32cb7e1f7468/volumes"
Sep 11 13:40:02.964606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.964511    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="871f39e3-4467-4172-8680-9469517b9fc9" path="/var/lib/kubelet/pods/871f39e3-4467-4172-8680-9469517b9fc9/volumes"
Sep 11 13:40:02.964855 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:02.964841    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="96e88044-00ed-4ea2-9157-b9454711bf29" path="/var/lib/kubelet/pods/96e88044-00ed-4ea2-9157-b9454711bf29/volumes"
Sep 11 13:40:04.808543 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.808502    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"]
Sep 11 13:40:04.809032 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809012    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="871f39e3-4467-4172-8680-9469517b9fc9" containerName="payload-processing"
Sep 11 13:40:04.809095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809036    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="871f39e3-4467-4172-8680-9469517b9fc9" containerName="payload-processing"
Sep 11 13:40:04.809095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809050    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="6255d6ef-3277-4547-8c8e-32cb7e1f7468" containerName="revoke-keys"
Sep 11 13:40:04.809095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809060    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="6255d6ef-3277-4547-8c8e-32cb7e1f7468" containerName="revoke-keys"
Sep 11 13:40:04.809095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809076    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="96e88044-00ed-4ea2-9157-b9454711bf29" containerName="payload-pre-processing"
Sep 11 13:40:04.809095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809085    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="96e88044-00ed-4ea2-9157-b9454711bf29" containerName="payload-pre-processing"
Sep 11 13:40:04.809430 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809165    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="871f39e3-4467-4172-8680-9469517b9fc9" containerName="payload-processing"
Sep 11 13:40:04.809430 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809178    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="96e88044-00ed-4ea2-9157-b9454711bf29" containerName="payload-pre-processing"
Sep 11 13:40:04.809430 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.809189    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="6255d6ef-3277-4547-8c8e-32cb7e1f7468" containerName="revoke-keys"
Sep 11 13:40:04.813658 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.813612    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:04.815850 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.815827    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:40:04.815966 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.815832    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:40:04.821105 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.821073    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"]
Sep 11 13:40:04.918599 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:04.918564    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8\" (UID: \"d25e2da4-58f9-4563-88fe-3baa634ae227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:05.019698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:05.019662    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8\" (UID: \"d25e2da4-58f9-4563-88fe-3baa634ae227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:05.020308 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:05.020287    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8\" (UID: \"d25e2da4-58f9-4563-88fe-3baa634ae227\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:05.125619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:05.125538    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:05.398874 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:05.398792    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"]
Sep 11 13:40:05.401793 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:05.401760    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd25e2da4_58f9_4563_88fe_3baa634ae227.slice/crio-b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113 WatchSource:0}: Error finding container b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113: Status 404 returned error can't find the container with id b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113
Sep 11 13:40:06.315419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:06.315330    2551 generic.go:356] "Generic (PLEG): container finished" podID="d25e2da4-58f9-4563-88fe-3baa634ae227" containerID="ca7d65834679c90aaf0f857d62c9ac89c6f26ade0ca4cd202041c4263e46a22f" exitCode=0
Sep 11 13:40:06.315419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:06.315404    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8" event={"ID":"d25e2da4-58f9-4563-88fe-3baa634ae227","Type":"ContainerDied","Data":"ca7d65834679c90aaf0f857d62c9ac89c6f26ade0ca4cd202041c4263e46a22f"}
Sep 11 13:40:06.315821 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:06.315440    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8" event={"ID":"d25e2da4-58f9-4563-88fe-3baa634ae227","Type":"ContainerStarted","Data":"b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113"}
Sep 11 13:40:07.453493 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:07.453464    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:07.541641 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:07.541596    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca\") pod \"d25e2da4-58f9-4563-88fe-3baa634ae227\" (UID: \"d25e2da4-58f9-4563-88fe-3baa634ae227\") "
Sep 11 13:40:07.541968 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:07.541945    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d25e2da4-58f9-4563-88fe-3baa634ae227" (UID: "d25e2da4-58f9-4563-88fe-3baa634ae227"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:07.642344 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:07.642261    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d25e2da4-58f9-4563-88fe-3baa634ae227-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:08.327006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:08.326972    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8" event={"ID":"d25e2da4-58f9-4563-88fe-3baa634ae227","Type":"ContainerDied","Data":"b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113"}
Sep 11 13:40:08.327006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:08.326992    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"
Sep 11 13:40:08.327189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:08.327020    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8b071d89780f7d94bf93653276c734bd74823edcefedfee4eeed60fe0176113"
Sep 11 13:40:14.350467 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.350427    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:40:14.353159 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.350723    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" podUID="b439611f-ba22-4bcc-8480-03db3fc9b38c" containerName="payload-processing" containerID="cri-o://8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6" gracePeriod=30
Sep 11 13:40:14.361614 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.361590    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:40:14.361851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.361808    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" podUID="a61ca4da-c59e-4e68-a9f3-168dfa552568" containerName="payload-pre-processing" containerID="cri-o://a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603" gracePeriod=30
Sep 11 13:40:14.598582 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.598555    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:40:14.600003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.599980    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"]
Sep 11 13:40:14.606531 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.606476    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-62b0dfd4-ff9z8"]
Sep 11 13:40:14.626887 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.626866    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:40:14.704262 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704236    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sf9kx\" (UniqueName: \"kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx\") pod \"b439611f-ba22-4bcc-8480-03db3fc9b38c\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") "
Sep 11 13:40:14.704455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704274    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume\") pod \"a61ca4da-c59e-4e68-a9f3-168dfa552568\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") "
Sep 11 13:40:14.704455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704304    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8\") pod \"a61ca4da-c59e-4e68-a9f3-168dfa552568\" (UID: \"a61ca4da-c59e-4e68-a9f3-168dfa552568\") "
Sep 11 13:40:14.704455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704341    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume\") pod \"b439611f-ba22-4bcc-8480-03db3fc9b38c\" (UID: \"b439611f-ba22-4bcc-8480-03db3fc9b38c\") "
Sep 11 13:40:14.704675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704623    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "a61ca4da-c59e-4e68-a9f3-168dfa552568" (UID: "a61ca4da-c59e-4e68-a9f3-168dfa552568"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:14.704769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.704748    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "b439611f-ba22-4bcc-8480-03db3fc9b38c" (UID: "b439611f-ba22-4bcc-8480-03db3fc9b38c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:40:14.706406 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.706384    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8" (OuterVolumeSpecName: "kube-api-access-28df8") pod "a61ca4da-c59e-4e68-a9f3-168dfa552568" (UID: "a61ca4da-c59e-4e68-a9f3-168dfa552568"). InnerVolumeSpecName "kube-api-access-28df8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:40:14.706503 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.706479    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx" (OuterVolumeSpecName: "kube-api-access-sf9kx") pod "b439611f-ba22-4bcc-8480-03db3fc9b38c" (UID: "b439611f-ba22-4bcc-8480-03db3fc9b38c"). InnerVolumeSpecName "kube-api-access-sf9kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:40:14.805229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.805191    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/a61ca4da-c59e-4e68-a9f3-168dfa552568-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:14.805229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.805225    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-28df8\" (UniqueName: \"kubernetes.io/projected/a61ca4da-c59e-4e68-a9f3-168dfa552568-kube-api-access-28df8\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:14.805229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.805238    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b439611f-ba22-4bcc-8480-03db3fc9b38c-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:14.805479 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.805252    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sf9kx\" (UniqueName: \"kubernetes.io/projected/b439611f-ba22-4bcc-8480-03db3fc9b38c-kube-api-access-sf9kx\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:40:14.966013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:14.965919    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d25e2da4-58f9-4563-88fe-3baa634ae227" path="/var/lib/kubelet/pods/d25e2da4-58f9-4563-88fe-3baa634ae227/volumes"
Sep 11 13:40:15.365341 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.365308    2551 generic.go:356] "Generic (PLEG): container finished" podID="b439611f-ba22-4bcc-8480-03db3fc9b38c" containerID="8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6" exitCode=0
Sep 11 13:40:15.365827 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.365363    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"
Sep 11 13:40:15.365827 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.365385    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" event={"ID":"b439611f-ba22-4bcc-8480-03db3fc9b38c","Type":"ContainerDied","Data":"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"}
Sep 11 13:40:15.365827 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.365437    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn" event={"ID":"b439611f-ba22-4bcc-8480-03db3fc9b38c","Type":"ContainerDied","Data":"0b0a2cafaf424c13268829e1c62bad74b35d6960c2aec2824de7c24151e414e4"}
Sep 11 13:40:15.365827 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.365464    2551 scope.go:117] "RemoveContainer" containerID="8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"
Sep 11 13:40:15.366768 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.366706    2551 generic.go:356] "Generic (PLEG): container finished" podID="a61ca4da-c59e-4e68-a9f3-168dfa552568" containerID="a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603" exitCode=0
Sep 11 13:40:15.366841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.366765    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"
Sep 11 13:40:15.366841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.366800    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" event={"ID":"a61ca4da-c59e-4e68-a9f3-168dfa552568","Type":"ContainerDied","Data":"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"}
Sep 11 13:40:15.366931 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.366855    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm" event={"ID":"a61ca4da-c59e-4e68-a9f3-168dfa552568","Type":"ContainerDied","Data":"d96acb85e7adf2728a6364ee80a8bb3906c14c159d6994fb9490cb623549ee07"}
Sep 11 13:40:15.374999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.374982    2551 scope.go:117] "RemoveContainer" containerID="8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"
Sep 11 13:40:15.375329 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:40:15.375308    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6\": container with ID starting with 8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6 not found: ID does not exist" containerID="8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"
Sep 11 13:40:15.375396 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.375340    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6"} err="failed to get container status \"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6\": rpc error: code = NotFound desc = could not find container \"8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6\": container with ID starting with 8dda4d0384a1c55e7a84441415db984dda1c323c63236c250efa0cf098cdb2a6 not found: ID does not exist"
Sep 11 13:40:15.375396 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.375356    2551 scope.go:117] "RemoveContainer" containerID="a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"
Sep 11 13:40:15.383295 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.383279    2551 scope.go:117] "RemoveContainer" containerID="a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"
Sep 11 13:40:15.383560 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:40:15.383543    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603\": container with ID starting with a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603 not found: ID does not exist" containerID="a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"
Sep 11 13:40:15.383642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.383569    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603"} err="failed to get container status \"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603\": rpc error: code = NotFound desc = could not find container \"a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603\": container with ID starting with a8ae9967a1c1b540aee8318ca63611fbaf218571cead7e3eb5f3b905219fa603 not found: ID does not exist"
Sep 11 13:40:15.412834 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.412808    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:40:15.424721 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.424695    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-62b0dfd4-5d89bf678f-rgzwn"]
Sep 11 13:40:15.439361 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.439329    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:40:15.454579 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:15.454543    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-62b0dfd4-5d5fc6d5db-6jdqm"]
Sep 11 13:40:16.692439 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692400    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692735    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="a61ca4da-c59e-4e68-a9f3-168dfa552568" containerName="payload-pre-processing"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692747    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="a61ca4da-c59e-4e68-a9f3-168dfa552568" containerName="payload-pre-processing"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692754    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d25e2da4-58f9-4563-88fe-3baa634ae227" containerName="revoke-keys"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692761    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d25e2da4-58f9-4563-88fe-3baa634ae227" containerName="revoke-keys"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692772    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b439611f-ba22-4bcc-8480-03db3fc9b38c" containerName="payload-processing"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692778    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b439611f-ba22-4bcc-8480-03db3fc9b38c" containerName="payload-processing"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692824    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b439611f-ba22-4bcc-8480-03db3fc9b38c" containerName="payload-processing"
Sep 11 13:40:16.692831 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692833    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d25e2da4-58f9-4563-88fe-3baa634ae227" containerName="revoke-keys"
Sep 11 13:40:16.693122 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.692840    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="a61ca4da-c59e-4e68-a9f3-168dfa552568" containerName="payload-pre-processing"
Sep 11 13:40:16.697159 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.697139    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.699426 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.699401    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-3dd80c\""
Sep 11 13:40:16.699547 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.699461    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-3dd80c-dockercfg-798ps\""
Sep 11 13:40:16.705200 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.705174    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:40:16.783648 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.783593    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:40:16.787052 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.787031    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.800533 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.800503    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:40:16.821796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.821772    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cq6vp\" (UniqueName: \"kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.821952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.821813    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hlm7k\" (UniqueName: \"kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.821952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.821876    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.821952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.821909    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.922955 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.922911    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hlm7k\" (UniqueName: \"kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.923155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.922987    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.923155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.923012    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.923155 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.923062    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cq6vp\" (UniqueName: \"kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.923777 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.923757    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.923850 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.923780    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.932227 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.932203    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-cq6vp\" (UniqueName: \"kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp\") pod \"payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:16.932673 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.932654    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-hlm7k\" (UniqueName: \"kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k\") pod \"payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:16.965228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.965152    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="a61ca4da-c59e-4e68-a9f3-168dfa552568" path="/var/lib/kubelet/pods/a61ca4da-c59e-4e68-a9f3-168dfa552568/volumes"
Sep 11 13:40:16.965697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:16.965681    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b439611f-ba22-4bcc-8480-03db3fc9b38c" path="/var/lib/kubelet/pods/b439611f-ba22-4bcc-8480-03db3fc9b38c/volumes"
Sep 11 13:40:17.008014 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.007978    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:17.097061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.097030    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:17.283836 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.283812    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:40:17.285907 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:17.285878    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod76cad1b3_71f5_430e_9027_d4708c8416a9.slice/crio-d5440f73dc4fd589721c01557bb68c65d39bbe9cf1009b0366d9abe15a428072 WatchSource:0}: Error finding container d5440f73dc4fd589721c01557bb68c65d39bbe9cf1009b0366d9abe15a428072: Status 404 returned error can't find the container with id d5440f73dc4fd589721c01557bb68c65d39bbe9cf1009b0366d9abe15a428072
Sep 11 13:40:17.377623 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.377590    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:40:17.389282 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.389232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" event={"ID":"76cad1b3-71f5-430e-9027-d4708c8416a9","Type":"ContainerStarted","Data":"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"}
Sep 11 13:40:17.389412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.389339    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" event={"ID":"76cad1b3-71f5-430e-9027-d4708c8416a9","Type":"ContainerStarted","Data":"d5440f73dc4fd589721c01557bb68c65d39bbe9cf1009b0366d9abe15a428072"}
Sep 11 13:40:17.389491 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:17.389469    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:18.395586 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:18.395551    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" event={"ID":"66684dc5-1415-4550-bb36-af0b1a495906","Type":"ContainerStarted","Data":"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"}
Sep 11 13:40:18.396023 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:18.395611    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" event={"ID":"66684dc5-1415-4550-bb36-af0b1a495906","Type":"ContainerStarted","Data":"7c76ce9a50053ad073026b9e1ff50c233e64372f3a4c74ad36f5548baa9ab7bf"}
Sep 11 13:40:18.396023 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:18.395814    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:18.415475 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:18.415424    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" podStartSLOduration=2.415407832 podStartE2EDuration="2.415407832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:17.414540188 +0000 UTC m=+1248.915444661" watchObservedRunningTime="2026-09-11 13:40:18.415407832 +0000 UTC m=+1249.916312281"
Sep 11 13:40:18.415839 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:18.415812    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" podStartSLOduration=2.415805155 podStartE2EDuration="2.415805155s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:18.413930576 +0000 UTC m=+1249.914835045" watchObservedRunningTime="2026-09-11 13:40:18.415805155 +0000 UTC m=+1249.916709603"
Sep 11 13:40:28.398279 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:28.398204    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:40:29.402662 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:29.402549    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:40:43.024208 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.024172    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:40:43.028848 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.028827    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.031670 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.031649    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w2-7bfa93-dockercfg-swmnw\""
Sep 11 13:40:43.031779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.031651    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w2-7bfa93\""
Sep 11 13:40:43.042379 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.042352    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:40:43.115323 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.115288    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:40:43.119044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.119028    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.131316 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.131289    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:40:43.143376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.143350    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.143517 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.143408    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sgbr5\" (UniqueName: \"kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.243948 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.243916    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sgbr5\" (UniqueName: \"kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.244124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.243991    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pdg27\" (UniqueName: \"kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.244124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.244017    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.244124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.244035    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.244697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.244678    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.254252 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.254219    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-sgbr5\" (UniqueName: \"kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5\") pod \"payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.339211 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.339131    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:43.345393 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.345366    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pdg27\" (UniqueName: \"kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.345522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.345398    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.346239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.346210    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.376312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.376281    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-pdg27\" (UniqueName: \"kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27\") pod \"payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.430037 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.430001    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:43.621266 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.621183    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:40:43.624517 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:43.624486    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa61f7dd_b7d6_452f_aa62_2989e19ead01.slice/crio-84290f64ea43f8d8b5ecca55c4da5a2bbcb361afc816a7439b5473e0e58bf409 WatchSource:0}: Error finding container 84290f64ea43f8d8b5ecca55c4da5a2bbcb361afc816a7439b5473e0e58bf409: Status 404 returned error can't find the container with id 84290f64ea43f8d8b5ecca55c4da5a2bbcb361afc816a7439b5473e0e58bf409
Sep 11 13:40:43.724796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:43.724768    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:40:43.726635 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:43.726590    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod774ccd65_5059_42c3_92be_1b6472cfaa3c.slice/crio-a256c18c631feede0d2b28ab388d365afd456c6f575def1479dd9bcbe8c5d8da WatchSource:0}: Error finding container a256c18c631feede0d2b28ab388d365afd456c6f575def1479dd9bcbe8c5d8da: Status 404 returned error can't find the container with id a256c18c631feede0d2b28ab388d365afd456c6f575def1479dd9bcbe8c5d8da
Sep 11 13:40:44.527501 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.527467    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" event={"ID":"774ccd65-5059-42c3-92be-1b6472cfaa3c","Type":"ContainerStarted","Data":"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"}
Sep 11 13:40:44.528018 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.527526    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" event={"ID":"774ccd65-5059-42c3-92be-1b6472cfaa3c","Type":"ContainerStarted","Data":"a256c18c631feede0d2b28ab388d365afd456c6f575def1479dd9bcbe8c5d8da"}
Sep 11 13:40:44.528018 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.527571    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:44.529117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.529096    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" event={"ID":"fa61f7dd-b7d6-452f-aa62-2989e19ead01","Type":"ContainerStarted","Data":"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"}
Sep 11 13:40:44.529205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.529140    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" event={"ID":"fa61f7dd-b7d6-452f-aa62-2989e19ead01","Type":"ContainerStarted","Data":"84290f64ea43f8d8b5ecca55c4da5a2bbcb361afc816a7439b5473e0e58bf409"}
Sep 11 13:40:44.529205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.529160    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:44.551843 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.551787    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" podStartSLOduration=1.551766766 podStartE2EDuration="1.551766766s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:44.548843788 +0000 UTC m=+1276.049748260" watchObservedRunningTime="2026-09-11 13:40:44.551766766 +0000 UTC m=+1276.052671212"
Sep 11 13:40:44.575510 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:44.575453    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" podStartSLOduration=2.575431997 podStartE2EDuration="2.575431997s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:44.575101496 +0000 UTC m=+1276.076005946" watchObservedRunningTime="2026-09-11 13:40:44.575431997 +0000 UTC m=+1276.076336447"
Sep 11 13:40:47.750221 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.750186    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:40:47.753761 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.753738    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:47.756003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.755981    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-d6ca2b\""
Sep 11 13:40:47.766762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.766736    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:40:47.823045 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.823015    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:40:47.826471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.826450    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:47.828795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.828776    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-d6ca2b-dockercfg-kt9p4\""
Sep 11 13:40:47.837382 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.837360    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:40:47.900934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.900898    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mdn7v\" (UniqueName: \"kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:47.901093 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:47.900944    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.002477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.002393    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mdn7v\" (UniqueName: \"kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.002477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.002434    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.002477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.002459    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.002754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.002508    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4wdv\" (UniqueName: \"kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.003120 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.003102    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.012382 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.012353    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-mdn7v\" (UniqueName: \"kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v\") pod \"payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.064206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.064166    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.103796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.103757    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.103990 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.103832    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4wdv\" (UniqueName: \"kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.104480 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.104461    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.114774 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.114751    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4wdv\" (UniqueName: \"kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv\") pod \"payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.137038 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.137009    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.343552 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.343524    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:40:48.345115 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:48.345089    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd5ad2cb2_94cb_43c0_8f2e_d822d70549cf.slice/crio-c48b20d2b2129a6472d62538a89cd7d88c83d5f0c4cd44286e06a77f53782695 WatchSource:0}: Error finding container c48b20d2b2129a6472d62538a89cd7d88c83d5f0c4cd44286e06a77f53782695: Status 404 returned error can't find the container with id c48b20d2b2129a6472d62538a89cd7d88c83d5f0c4cd44286e06a77f53782695
Sep 11 13:40:48.424331 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.424297    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:40:48.427508 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:40:48.427480    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd40b9629_03af_494a_83e8_e4bd04a2e66d.slice/crio-00647a14d65d01b6a002afe04f590afe6eb2ed534a01f8653a11c52f297950ca WatchSource:0}: Error finding container 00647a14d65d01b6a002afe04f590afe6eb2ed534a01f8653a11c52f297950ca: Status 404 returned error can't find the container with id 00647a14d65d01b6a002afe04f590afe6eb2ed534a01f8653a11c52f297950ca
Sep 11 13:40:48.550720 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.550681    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" event={"ID":"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf","Type":"ContainerStarted","Data":"329ea169fb8e7fe2f4bde8a7a4098e1a10ed98514ea5c39e5b8ed364f3ef8875"}
Sep 11 13:40:48.550907 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.550749    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" event={"ID":"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf","Type":"ContainerStarted","Data":"c48b20d2b2129a6472d62538a89cd7d88c83d5f0c4cd44286e06a77f53782695"}
Sep 11 13:40:48.550907 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.550788    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:48.553212 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.553177    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" event={"ID":"d40b9629-03af-494a-83e8-e4bd04a2e66d","Type":"ContainerStarted","Data":"6bb503fef0f06d839b254a726ec6f8e6a3c90ebca1b1c012f27e35fb0f82fa01"}
Sep 11 13:40:48.553340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.553232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" event={"ID":"d40b9629-03af-494a-83e8-e4bd04a2e66d","Type":"ContainerStarted","Data":"00647a14d65d01b6a002afe04f590afe6eb2ed534a01f8653a11c52f297950ca"}
Sep 11 13:40:48.553340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.553276    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:40:48.570579 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.570532    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" podStartSLOduration=1.570515709 podStartE2EDuration="1.570515709s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:48.567455485 +0000 UTC m=+1280.068359935" watchObservedRunningTime="2026-09-11 13:40:48.570515709 +0000 UTC m=+1280.071420157"
Sep 11 13:40:48.585243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:48.585194    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" podStartSLOduration=1.5851813030000002 podStartE2EDuration="1.585181303s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:40:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:40:48.583559324 +0000 UTC m=+1280.084463803" watchObservedRunningTime="2026-09-11 13:40:48.585181303 +0000 UTC m=+1280.086085779"
Sep 11 13:40:55.536493 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:55.536464    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:40:55.536976 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:55.536534    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:40:59.561471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:59.561438    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:40:59.561891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:40:59.561509    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:41:12.429971 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.429939    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:41:12.433317 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.433300    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.435540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.435509    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w2-32b673\""
Sep 11 13:41:12.448528 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.448497    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:41:12.458751 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.458726    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:41:12.462206 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.462189    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.464465 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.464449    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w2-32b673-dockercfg-p7ntd\""
Sep 11 13:41:12.474528 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.474505    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:41:12.516769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.516738    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.516900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.516783    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nb5\" (UniqueName: \"kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.516900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.516851    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-22cl2\" (UniqueName: \"kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.516900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.516871    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.617878 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.617836    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-22cl2\" (UniqueName: \"kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.617878 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.617878    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.618106 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.617929    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.618106 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.617962    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nb5\" (UniqueName: \"kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.618715 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.618697    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.618764 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.618714    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.626974 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.626947    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-22cl2\" (UniqueName: \"kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2\") pod \"payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:12.627089 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.627042    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nb5\" (UniqueName: \"kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5\") pod \"payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.743746 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.743713    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:12.774323 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:12.774288    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:13.021083 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.021057    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:41:13.023356 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:41:13.023322    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf5e7252_1c6e_437f_9378_69c6cf87fe30.slice/crio-35d30bb45b62e90fbdba6903b533e8d3a50685ec6551aac2bf912d0410bc1c77 WatchSource:0}: Error finding container 35d30bb45b62e90fbdba6903b533e8d3a50685ec6551aac2bf912d0410bc1c77: Status 404 returned error can't find the container with id 35d30bb45b62e90fbdba6903b533e8d3a50685ec6551aac2bf912d0410bc1c77
Sep 11 13:41:13.059411 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.059384    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:41:13.072704 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:41:13.072679    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod402d92e9_acd7_4aa0_b168_f18bd09fa187.slice/crio-e744fabcd380dd86245681da11aec2da24fc8cb9d3b154fdc06ec3654fae5ab8 WatchSource:0}: Error finding container e744fabcd380dd86245681da11aec2da24fc8cb9d3b154fdc06ec3654fae5ab8: Status 404 returned error can't find the container with id e744fabcd380dd86245681da11aec2da24fc8cb9d3b154fdc06ec3654fae5ab8
Sep 11 13:41:13.708581 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.708534    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" event={"ID":"cf5e7252-1c6e-437f-9378-69c6cf87fe30","Type":"ContainerStarted","Data":"618d2930c4ede3e02580c3c78e07473bb894a2e32d78dd124da622db90790a8e"}
Sep 11 13:41:13.709053 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.708657    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" event={"ID":"cf5e7252-1c6e-437f-9378-69c6cf87fe30","Type":"ContainerStarted","Data":"35d30bb45b62e90fbdba6903b533e8d3a50685ec6551aac2bf912d0410bc1c77"}
Sep 11 13:41:13.709053 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.708688    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:13.710218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.710191    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" event={"ID":"402d92e9-acd7-4aa0-b168-f18bd09fa187","Type":"ContainerStarted","Data":"1282efb033e7440ee27cfafd891336089e92c063220d80931f0345291ac01a46"}
Sep 11 13:41:13.710371 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.710232    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" event={"ID":"402d92e9-acd7-4aa0-b168-f18bd09fa187","Type":"ContainerStarted","Data":"e744fabcd380dd86245681da11aec2da24fc8cb9d3b154fdc06ec3654fae5ab8"}
Sep 11 13:41:13.710371 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.710313    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:13.726893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.726833    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" podStartSLOduration=1.726813206 podStartE2EDuration="1.726813206s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:41:13.724253821 +0000 UTC m=+1305.225158562" watchObservedRunningTime="2026-09-11 13:41:13.726813206 +0000 UTC m=+1305.227717658"
Sep 11 13:41:13.741683 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:13.741612    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" podStartSLOduration=1.741595975 podStartE2EDuration="1.741595975s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:41:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:41:13.739768109 +0000 UTC m=+1305.240672584" watchObservedRunningTime="2026-09-11 13:41:13.741595975 +0000 UTC m=+1305.242500761"
Sep 11 13:41:24.717539 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:24.717508    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:41:24.718150 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:24.717667    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:41:52.324499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.324408    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:41:52.328339 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.328315    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.330768 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.330748    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-ceb67528\""
Sep 11 13:41:52.331142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.331126    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-ceb67528-dockercfg-cckkl\""
Sep 11 13:41:52.344232 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.344206    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:41:52.354153 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.354118    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.354373 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.354201    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74fbg\" (UniqueName: \"kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.410473 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.410439    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:41:52.414336 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.414312    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.425825 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.425794    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:41:52.455139 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.455096    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74fbg\" (UniqueName: \"kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.455139 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.455141    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x28bp\" (UniqueName: \"kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.455384 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.455196    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.455384 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.455291    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.456044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.456013    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.463663 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.463644    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-74fbg\" (UniqueName: \"kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg\") pod \"payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.556494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.556456    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.556703 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.556539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x28bp\" (UniqueName: \"kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.557408 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.557382    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.564674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.564651    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-x28bp\" (UniqueName: \"kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp\") pod \"payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") " pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.638669 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.638555    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:52.728431 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.728397    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:52.917956 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:52.917933    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:41:52.920445 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:41:52.920416    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5487b9b7_df68_42b3_b550_ebefd24db197.slice/crio-7fe8ee03f3493520bd936ff3c107cceaaaaefbd02f7a1f53ab4e9b8832abb0c3 WatchSource:0}: Error finding container 7fe8ee03f3493520bd936ff3c107cceaaaaefbd02f7a1f53ab4e9b8832abb0c3: Status 404 returned error can't find the container with id 7fe8ee03f3493520bd936ff3c107cceaaaaefbd02f7a1f53ab4e9b8832abb0c3
Sep 11 13:41:53.008750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.008727    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:41:53.011499 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:41:53.011467    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf96aeda_ab70_45df_a94f_cf9c2b166f9f.slice/crio-b41374c58c4e6a77a66ace9467359cb9fe19ad3a79992febb6bc6e872e81da3f WatchSource:0}: Error finding container b41374c58c4e6a77a66ace9467359cb9fe19ad3a79992febb6bc6e872e81da3f: Status 404 returned error can't find the container with id b41374c58c4e6a77a66ace9467359cb9fe19ad3a79992febb6bc6e872e81da3f
Sep 11 13:41:53.918199 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.918158    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" event={"ID":"bf96aeda-ab70-45df-a94f-cf9c2b166f9f","Type":"ContainerStarted","Data":"061f371077a8274df0fbf523b6550cb7026cf121c1967be7cbee8b113e8179a6"}
Sep 11 13:41:53.918606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.918219    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" event={"ID":"bf96aeda-ab70-45df-a94f-cf9c2b166f9f","Type":"ContainerStarted","Data":"b41374c58c4e6a77a66ace9467359cb9fe19ad3a79992febb6bc6e872e81da3f"}
Sep 11 13:41:53.918606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.918291    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:41:53.919872 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.919846    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" event={"ID":"5487b9b7-df68-42b3-b550-ebefd24db197","Type":"ContainerStarted","Data":"7aafb3e5bf3c7e24f69b2a5672bc4ea50f9c6fcea48be02ca6277a2634dbf34d"}
Sep 11 13:41:53.920011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.919887    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" event={"ID":"5487b9b7-df68-42b3-b550-ebefd24db197","Type":"ContainerStarted","Data":"7fe8ee03f3493520bd936ff3c107cceaaaaefbd02f7a1f53ab4e9b8832abb0c3"}
Sep 11 13:41:53.920011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.919942    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:41:53.935733 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.935688    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" podStartSLOduration=1.9356727390000001 podStartE2EDuration="1.935672739s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:41:53.934145949 +0000 UTC m=+1345.435050420" watchObservedRunningTime="2026-09-11 13:41:53.935672739 +0000 UTC m=+1345.436577187"
Sep 11 13:41:53.949616 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:41:53.949570    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" podStartSLOduration=1.949554974 podStartE2EDuration="1.949554974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:41:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:41:53.948508501 +0000 UTC m=+1345.449412950" watchObservedRunningTime="2026-09-11 13:41:53.949554974 +0000 UTC m=+1345.450459426"
Sep 11 13:42:04.926575 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:42:04.926543    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:42:04.927062 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:42:04.926612    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:42:29.121539 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:42:29.121507    2551 scope.go:117] "RemoveContainer" containerID="9c980da8e71ffe1701eeaafcfdcbae4f844977315b377feeede7496116dd6c61"
Sep 11 13:42:29.130331 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:42:29.130307    2551 scope.go:117] "RemoveContainer" containerID="616b386d4dcf26521741e08fe21a3c2a732b441367339c20ce63e942b17a0ee3"
Sep 11 13:43:12.639848 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.639815    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"]
Sep 11 13:43:12.643353 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.643336    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:12.647334 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.647309    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:43:12.647464 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.647420    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:43:12.669538 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.669509    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"]
Sep 11 13:43:12.775086 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.775050    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw\" (UID: \"98eea105-b5c5-46ba-898e-9480e3c41cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:12.876507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.876469    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw\" (UID: \"98eea105-b5c5-46ba-898e-9480e3c41cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:12.877162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.877130    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw\" (UID: \"98eea105-b5c5-46ba-898e-9480e3c41cc8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:12.953430 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:12.953344    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:13.230898 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:13.230865    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"]
Sep 11 13:43:13.233870 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:43:13.233831    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod98eea105_b5c5_46ba_898e_9480e3c41cc8.slice/crio-8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb WatchSource:0}: Error finding container 8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb: Status 404 returned error can't find the container with id 8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb
Sep 11 13:43:13.334514 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:13.334482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw" event={"ID":"98eea105-b5c5-46ba-898e-9480e3c41cc8","Type":"ContainerStarted","Data":"8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb"}
Sep 11 13:43:14.340714 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:14.340676    2551 generic.go:356] "Generic (PLEG): container finished" podID="98eea105-b5c5-46ba-898e-9480e3c41cc8" containerID="2b7285ed8fbeb76d36c020b956cc4ba4a6befea17edc5970a502a31807b2488e" exitCode=0
Sep 11 13:43:14.341143 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:14.340724    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw" event={"ID":"98eea105-b5c5-46ba-898e-9480e3c41cc8","Type":"ContainerDied","Data":"2b7285ed8fbeb76d36c020b956cc4ba4a6befea17edc5970a502a31807b2488e"}
Sep 11 13:43:15.486172 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:15.486142    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:15.602304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:15.602205    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca\") pod \"98eea105-b5c5-46ba-898e-9480e3c41cc8\" (UID: \"98eea105-b5c5-46ba-898e-9480e3c41cc8\") "
Sep 11 13:43:15.602618 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:15.602589    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "98eea105-b5c5-46ba-898e-9480e3c41cc8" (UID: "98eea105-b5c5-46ba-898e-9480e3c41cc8"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:43:15.702932 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:15.702896    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/98eea105-b5c5-46ba-898e-9480e3c41cc8-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:43:16.354034 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:16.354006    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"
Sep 11 13:43:16.354218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:16.354038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw" event={"ID":"98eea105-b5c5-46ba-898e-9480e3c41cc8","Type":"ContainerDied","Data":"8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb"}
Sep 11 13:43:16.354218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:16.354122    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a3c5c1e5893b7bd2c38f5070a6439af7fd2294b021f5c6da69a89c59c7f2ecb"
Sep 11 13:43:29.154545 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.154322    2551 scope.go:117] "RemoveContainer" containerID="e638de9da3060393dfe7a73d97f90435f79adfbb8810cbd7b0a9748316dbe439"
Sep 11 13:43:29.166994 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.166966    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:43:29.167250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.167194    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" podUID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" containerName="payload-processing" containerID="cri-o://061f371077a8274df0fbf523b6550cb7026cf121c1967be7cbee8b113e8179a6" gracePeriod=30
Sep 11 13:43:29.169616 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.169590    2551 scope.go:117] "RemoveContainer" containerID="db74a6260b38722d7774475748c1a7ec02b2aba89a21b46f227b97873f1175ac"
Sep 11 13:43:29.181027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.180954    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:43:29.181272 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.181207    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" podUID="5487b9b7-df68-42b3-b550-ebefd24db197" containerName="payload-pre-processing" containerID="cri-o://7aafb3e5bf3c7e24f69b2a5672bc4ea50f9c6fcea48be02ca6277a2634dbf34d" gracePeriod=30
Sep 11 13:43:29.424698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.424601    2551 generic.go:356] "Generic (PLEG): container finished" podID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" containerID="061f371077a8274df0fbf523b6550cb7026cf121c1967be7cbee8b113e8179a6" exitCode=0
Sep 11 13:43:29.424698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.424673    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" event={"ID":"bf96aeda-ab70-45df-a94f-cf9c2b166f9f","Type":"ContainerDied","Data":"061f371077a8274df0fbf523b6550cb7026cf121c1967be7cbee8b113e8179a6"}
Sep 11 13:43:29.426486 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.426459    2551 generic.go:356] "Generic (PLEG): container finished" podID="5487b9b7-df68-42b3-b550-ebefd24db197" containerID="7aafb3e5bf3c7e24f69b2a5672bc4ea50f9c6fcea48be02ca6277a2634dbf34d" exitCode=0
Sep 11 13:43:29.426621 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.426554    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" event={"ID":"5487b9b7-df68-42b3-b550-ebefd24db197","Type":"ContainerDied","Data":"7aafb3e5bf3c7e24f69b2a5672bc4ea50f9c6fcea48be02ca6277a2634dbf34d"}
Sep 11 13:43:29.469833 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.469806    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:43:29.472522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.472502    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:43:29.523697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.523654    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume\") pod \"5487b9b7-df68-42b3-b550-ebefd24db197\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") "
Sep 11 13:43:29.523860 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.523713    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-74fbg\" (UniqueName: \"kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg\") pod \"5487b9b7-df68-42b3-b550-ebefd24db197\" (UID: \"5487b9b7-df68-42b3-b550-ebefd24db197\") "
Sep 11 13:43:29.523860 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.523803    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-x28bp\" (UniqueName: \"kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp\") pod \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") "
Sep 11 13:43:29.523860 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.523858    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume\") pod \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\" (UID: \"bf96aeda-ab70-45df-a94f-cf9c2b166f9f\") "
Sep 11 13:43:29.524068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.524036    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "5487b9b7-df68-42b3-b550-ebefd24db197" (UID: "5487b9b7-df68-42b3-b550-ebefd24db197"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:43:29.524315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.524287    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "bf96aeda-ab70-45df-a94f-cf9c2b166f9f" (UID: "bf96aeda-ab70-45df-a94f-cf9c2b166f9f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:43:29.526006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.525975    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp" (OuterVolumeSpecName: "kube-api-access-x28bp") pod "bf96aeda-ab70-45df-a94f-cf9c2b166f9f" (UID: "bf96aeda-ab70-45df-a94f-cf9c2b166f9f"). InnerVolumeSpecName "kube-api-access-x28bp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:43:29.526099 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.525981    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg" (OuterVolumeSpecName: "kube-api-access-74fbg") pod "5487b9b7-df68-42b3-b550-ebefd24db197" (UID: "5487b9b7-df68-42b3-b550-ebefd24db197"). InnerVolumeSpecName "kube-api-access-74fbg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:43:29.547870 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.547841    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"]
Sep 11 13:43:29.550728 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.550700    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-ceb67528-p4sxw"]
Sep 11 13:43:29.624493 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.624463    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:43:29.624493 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.624486    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5487b9b7-df68-42b3-b550-ebefd24db197-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:43:29.624493 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.624497    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74fbg\" (UniqueName: \"kubernetes.io/projected/5487b9b7-df68-42b3-b550-ebefd24db197-kube-api-access-74fbg\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:43:29.624736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:29.624506    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x28bp\" (UniqueName: \"kubernetes.io/projected/bf96aeda-ab70-45df-a94f-cf9c2b166f9f-kube-api-access-x28bp\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:43:30.432859 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.432829    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"
Sep 11 13:43:30.433302 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.432872    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76" event={"ID":"5487b9b7-df68-42b3-b550-ebefd24db197","Type":"ContainerDied","Data":"7fe8ee03f3493520bd936ff3c107cceaaaaefbd02f7a1f53ab4e9b8832abb0c3"}
Sep 11 13:43:30.433302 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.432940    2551 scope.go:117] "RemoveContainer" containerID="7aafb3e5bf3c7e24f69b2a5672bc4ea50f9c6fcea48be02ca6277a2634dbf34d"
Sep 11 13:43:30.434245 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.434216    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5" event={"ID":"bf96aeda-ab70-45df-a94f-cf9c2b166f9f","Type":"ContainerDied","Data":"b41374c58c4e6a77a66ace9467359cb9fe19ad3a79992febb6bc6e872e81da3f"}
Sep 11 13:43:30.434320 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.434255    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"
Sep 11 13:43:30.442514 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.442496    2551 scope.go:117] "RemoveContainer" containerID="061f371077a8274df0fbf523b6550cb7026cf121c1967be7cbee8b113e8179a6"
Sep 11 13:43:30.457218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.457189    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:43:30.459544 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.459520    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-ceb67528-796cb6588b-mxb76"]
Sep 11 13:43:30.469873 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.469848    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:43:30.477584 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.477561    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-ceb67528-7774c4fb6f-n22w5"]
Sep 11 13:43:30.966246 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.966214    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="5487b9b7-df68-42b3-b550-ebefd24db197" path="/var/lib/kubelet/pods/5487b9b7-df68-42b3-b550-ebefd24db197/volumes"
Sep 11 13:43:30.966540 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.966528    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="98eea105-b5c5-46ba-898e-9480e3c41cc8" path="/var/lib/kubelet/pods/98eea105-b5c5-46ba-898e-9480e3c41cc8/volumes"
Sep 11 13:43:30.966897 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:43:30.966884    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" path="/var/lib/kubelet/pods/bf96aeda-ab70-45df-a94f-cf9c2b166f9f/volumes"
Sep 11 13:44:25.358172 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358139    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358506    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" containerName="payload-processing"
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358518    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" containerName="payload-processing"
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358528    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="98eea105-b5c5-46ba-898e-9480e3c41cc8" containerName="revoke-keys"
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358534    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="98eea105-b5c5-46ba-898e-9480e3c41cc8" containerName="revoke-keys"
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358542    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="5487b9b7-df68-42b3-b550-ebefd24db197" containerName="payload-pre-processing"
Sep 11 13:44:25.358569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358548    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="5487b9b7-df68-42b3-b550-ebefd24db197" containerName="payload-pre-processing"
Sep 11 13:44:25.358802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358603    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="98eea105-b5c5-46ba-898e-9480e3c41cc8" containerName="revoke-keys"
Sep 11 13:44:25.358802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358613    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bf96aeda-ab70-45df-a94f-cf9c2b166f9f" containerName="payload-processing"
Sep 11 13:44:25.358802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.358621    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="5487b9b7-df68-42b3-b550-ebefd24db197" containerName="payload-pre-processing"
Sep 11 13:44:25.361332 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.361314    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.379699 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.379667    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd80c\"/\"sub-test-model-3dd80c-kserve-self-signed-certs\""
Sep 11 13:44:25.379847 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.379830    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd80c\"/\"kube-root-ca.crt\""
Sep 11 13:44:25.380062 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.380049    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd80c\"/\"default-dockercfg-6qs9k\""
Sep 11 13:44:25.380197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.380184    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-a-w3-3dd80c\"/\"openshift-service-ca.crt\""
Sep 11 13:44:25.408476 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408443    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:44:25.408793 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408770    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.408893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408811    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.408893 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408838    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-485lc\" (UniqueName: \"kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.408997 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408896    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.408997 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.408948    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.409064 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.409026    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.509994 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.509958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510006    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510029    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510071    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-485lc\" (UniqueName: \"kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510123    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510173    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510456    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510651 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510496    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.510651 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.510521    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.512337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.512315    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.512679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.512663    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.518992 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.518967    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-485lc\" (UniqueName: \"kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc\") pod \"sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") " pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.670349 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.670265    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:25.949819 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:25.949773    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:44:25.951464 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:44:25.951412    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod385fdac2_c7d3_46ec_9374_9d6054e8830e.slice/crio-86f0e36df21ffef7077f4d9a3d0a5399b02605f89b75f4423c2cc7c5fcbc7bce WatchSource:0}: Error finding container 86f0e36df21ffef7077f4d9a3d0a5399b02605f89b75f4423c2cc7c5fcbc7bce: Status 404 returned error can't find the container with id 86f0e36df21ffef7077f4d9a3d0a5399b02605f89b75f4423c2cc7c5fcbc7bce
Sep 11 13:44:26.732187 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:26.732149    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" event={"ID":"385fdac2-c7d3-46ec-9374-9d6054e8830e","Type":"ContainerStarted","Data":"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"}
Sep 11 13:44:26.732561 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:26.732199    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" event={"ID":"385fdac2-c7d3-46ec-9374-9d6054e8830e","Type":"ContainerStarted","Data":"86f0e36df21ffef7077f4d9a3d0a5399b02605f89b75f4423c2cc7c5fcbc7bce"}
Sep 11 13:44:26.755690 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:26.755621    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" podStartSLOduration=1.755604328 podStartE2EDuration="1.755604328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:44:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:44:26.755043394 +0000 UTC m=+1498.255947844" watchObservedRunningTime="2026-09-11 13:44:26.755604328 +0000 UTC m=+1498.256508794"
Sep 11 13:44:29.294338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:29.294308    2551 scope.go:117] "RemoveContainer" containerID="40755a4ac7ac25089826ef44995af808272c1074d4427ca0c1175bbcc11ca4da"
Sep 11 13:44:35.671286 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:35.671243    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:35.671697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:35.671329    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:35.683788 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:35.683764    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:44:35.794874 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:44:35.794846    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:45:00.172583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.172506    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"]
Sep 11 13:45:00.176246 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.176226    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"
Sep 11 13:45:00.180607 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.180582    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:45:00.182660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.182618    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"]
Sep 11 13:45:00.186178 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.186159    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"
Sep 11 13:45:00.186292 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.186209    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"
Sep 11 13:45:00.191386 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.191360    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"]
Sep 11 13:45:00.196302 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.196280    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"
Sep 11 13:45:00.199174 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.199151    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"]
Sep 11 13:45:00.203086 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.203059    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"
Sep 11 13:45:00.207687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.206846    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"]
Sep 11 13:45:00.223005 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.222969    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"]
Sep 11 13:45:00.239757 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.239710    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"
Sep 11 13:45:00.482019 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.481995    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"]
Sep 11 13:45:00.484644 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:45:00.484605    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4ff2946f_3d89_40c4_aaf5_f5b8e4791006.slice/crio-f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301 WatchSource:0}: Error finding container f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301: Status 404 returned error can't find the container with id f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301
Sep 11 13:45:00.512225 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.512204    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"]
Sep 11 13:45:00.517748 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:45:00.517716    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb1add6b_a09a_45e6_a11b_90c50366e5dd.slice/crio-b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7 WatchSource:0}: Error finding container b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7: Status 404 returned error can't find the container with id b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7
Sep 11 13:45:00.548786 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.548757    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"]
Sep 11 13:45:00.550353 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:45:00.550329    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2d3c7897_17e0_4a65_bbac_96ff0dd0b299.slice/crio-e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21 WatchSource:0}: Error finding container e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21: Status 404 returned error can't find the container with id e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21
Sep 11 13:45:00.912326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.912290    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv" event={"ID":"2d3c7897-17e0-4a65-bbac-96ff0dd0b299","Type":"ContainerStarted","Data":"0801ab20c46caf568b60d0b2683586d06eaf40e86fd738a86571383093f35e38"}
Sep 11 13:45:00.912524 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.912357    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv" event={"ID":"2d3c7897-17e0-4a65-bbac-96ff0dd0b299","Type":"ContainerStarted","Data":"e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21"}
Sep 11 13:45:00.913560 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.913527    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq" event={"ID":"bb1add6b-a09a-45e6-a11b-90c50366e5dd","Type":"ContainerStarted","Data":"b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7"}
Sep 11 13:45:00.915022 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.914999    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9" event={"ID":"4ff2946f-3d89-40c4-aaf5-f5b8e4791006","Type":"ContainerStarted","Data":"573bfc5f8391fb68baa6bb9735a0a18266471ad12f0c4ba646d3de550085be58"}
Sep 11 13:45:00.915117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.915034    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9" event={"ID":"4ff2946f-3d89-40c4-aaf5-f5b8e4791006","Type":"ContainerStarted","Data":"f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301"}
Sep 11 13:45:00.932953 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:00.932905    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv" podStartSLOduration=0.932891881 podStartE2EDuration="932.891881ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:45:00.930211489 +0000 UTC m=+1532.431115955" watchObservedRunningTime="2026-09-11 13:45:00.932891881 +0000 UTC m=+1532.433796330"
Sep 11 13:45:01.924558 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.924521    2551 generic.go:356] "Generic (PLEG): container finished" podID="bb1add6b-a09a-45e6-a11b-90c50366e5dd" containerID="d81920cf40f66e6f04a55a62bee2e4161458fae0be152f1b839c843abdab4ba4" exitCode=0
Sep 11 13:45:01.925015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.924607    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq" event={"ID":"bb1add6b-a09a-45e6-a11b-90c50366e5dd","Type":"ContainerDied","Data":"d81920cf40f66e6f04a55a62bee2e4161458fae0be152f1b839c843abdab4ba4"}
Sep 11 13:45:01.926455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.926428    2551 generic.go:356] "Generic (PLEG): container finished" podID="4ff2946f-3d89-40c4-aaf5-f5b8e4791006" containerID="573bfc5f8391fb68baa6bb9735a0a18266471ad12f0c4ba646d3de550085be58" exitCode=0
Sep 11 13:45:01.926577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.926513    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9" event={"ID":"4ff2946f-3d89-40c4-aaf5-f5b8e4791006","Type":"ContainerDied","Data":"573bfc5f8391fb68baa6bb9735a0a18266471ad12f0c4ba646d3de550085be58"}
Sep 11 13:45:01.928289 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.928268    2551 generic.go:356] "Generic (PLEG): container finished" podID="2d3c7897-17e0-4a65-bbac-96ff0dd0b299" containerID="0801ab20c46caf568b60d0b2683586d06eaf40e86fd738a86571383093f35e38" exitCode=0
Sep 11 13:45:01.928398 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.928309    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv" event={"ID":"2d3c7897-17e0-4a65-bbac-96ff0dd0b299","Type":"ContainerDied","Data":"0801ab20c46caf568b60d0b2683586d06eaf40e86fd738a86571383093f35e38"}
Sep 11 13:45:01.953994 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:01.953928    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9" podStartSLOduration=1.953904479 podStartE2EDuration="1.953904479s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:45:00.950558595 +0000 UTC m=+1532.451463048" watchObservedRunningTime="2026-09-11 13:45:01.953904479 +0000 UTC m=+1533.454808930"
Sep 11 13:45:03.118663 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.118609    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"
Sep 11 13:45:03.121896 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.121871    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"
Sep 11 13:45:03.136728 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.136698    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"
Sep 11 13:45:03.939909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.939881    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"
Sep 11 13:45:03.939909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.939906    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv" event={"ID":"2d3c7897-17e0-4a65-bbac-96ff0dd0b299","Type":"ContainerDied","Data":"e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21"}
Sep 11 13:45:03.940176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.939957    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e1d4515c9e539b92fb4e6fef726169b8c12ab2c73a3559aa3e28f1e01b1ccf21"
Sep 11 13:45:03.941273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.941249    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq" event={"ID":"bb1add6b-a09a-45e6-a11b-90c50366e5dd","Type":"ContainerDied","Data":"b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7"}
Sep 11 13:45:03.941403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.941291    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b7eae106584444b1a9aa8c6fe4e234ae5eb795b12c2155c00505c8eced8cd2a7"
Sep 11 13:45:03.941403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.941257    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"
Sep 11 13:45:03.942613 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.942596    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"
Sep 11 13:45:03.942613 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.942599    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9" event={"ID":"4ff2946f-3d89-40c4-aaf5-f5b8e4791006","Type":"ContainerDied","Data":"f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301"}
Sep 11 13:45:03.942748 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:03.942653    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f483dbe0ab26b5203d9a1a3118d636a4e32857da934804fcabde57d3fc33e301"
Sep 11 13:45:29.324621 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:29.324587    2551 scope.go:117] "RemoveContainer" containerID="485108e1c0e4360de7ff674066b191d347e6104025a26b10b70a13437474bf3f"
Sep 11 13:45:37.768238 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768203    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768566    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2d3c7897-17e0-4a65-bbac-96ff0dd0b299" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768577    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2d3c7897-17e0-4a65-bbac-96ff0dd0b299" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768599    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4ff2946f-3d89-40c4-aaf5-f5b8e4791006" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768605    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4ff2946f-3d89-40c4-aaf5-f5b8e4791006" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768613    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="bb1add6b-a09a-45e6-a11b-90c50366e5dd" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768619    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="bb1add6b-a09a-45e6-a11b-90c50366e5dd" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768709    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4ff2946f-3d89-40c4-aaf5-f5b8e4791006" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768718    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2d3c7897-17e0-4a65-bbac-96ff0dd0b299" containerName="cleanup"
Sep 11 13:45:37.768723 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.768725    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="bb1add6b-a09a-45e6-a11b-90c50366e5dd" containerName="cleanup"
Sep 11 13:45:37.771909 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.771892    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.776004 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.775984    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-65a9ed-dockercfg-djtbs\""
Sep 11 13:45:37.776285 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.776268    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-65a9ed\""
Sep 11 13:45:37.789102 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.789077    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:45:37.844281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.844252    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:45:37.848465 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.848436    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:37.860771 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.860743    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9vrsk\" (UniqueName: \"kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.860902 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.860866    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.872568 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.872535    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:45:37.961612 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.961571    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9q4dl\" (UniqueName: \"kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:37.961801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.961702    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.961801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.961763    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:37.961801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.961796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9vrsk\" (UniqueName: \"kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.962404 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.962383    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:37.975785 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:37.975759    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9vrsk\" (UniqueName: \"kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk\") pod \"payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:38.063198 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.063109    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:38.063198 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.063197    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9q4dl\" (UniqueName: \"kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:38.063984 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.063959    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:38.074756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.074728    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9q4dl\" (UniqueName: \"kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl\") pod \"payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") " pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:38.082272 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.082249    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:38.160988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.160956    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:38.360328 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.360297    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:45:38.362300 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:45:38.362278    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podacb6968c_102c_45ed_9a95_809bccc7bb1c.slice/crio-b81aedaf60a2fe98536e2d46b4a8f27044f5780e78969417022b0630bd22e68b WatchSource:0}: Error finding container b81aedaf60a2fe98536e2d46b4a8f27044f5780e78969417022b0630bd22e68b: Status 404 returned error can't find the container with id b81aedaf60a2fe98536e2d46b4a8f27044f5780e78969417022b0630bd22e68b
Sep 11 13:45:38.440755 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:38.440725    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:45:38.441618 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:45:38.441591    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod31fecef4_0662_4ae9_870f_ca38227621e4.slice/crio-5a3c9b06c9735ebc2f3a6d0d86c42ccbd72e6d4384abdf2992b9cf76f32e89fe WatchSource:0}: Error finding container 5a3c9b06c9735ebc2f3a6d0d86c42ccbd72e6d4384abdf2992b9cf76f32e89fe: Status 404 returned error can't find the container with id 5a3c9b06c9735ebc2f3a6d0d86c42ccbd72e6d4384abdf2992b9cf76f32e89fe
Sep 11 13:45:39.121759 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.121719    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" event={"ID":"acb6968c-102c-45ed-9a95-809bccc7bb1c","Type":"ContainerStarted","Data":"138de6105cdebae71831bc138b636f5777603c24d92d60e1c93c101e538b3925"}
Sep 11 13:45:39.122238 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.121785    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" event={"ID":"acb6968c-102c-45ed-9a95-809bccc7bb1c","Type":"ContainerStarted","Data":"b81aedaf60a2fe98536e2d46b4a8f27044f5780e78969417022b0630bd22e68b"}
Sep 11 13:45:39.122238 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.121863    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:39.123315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.123289    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" event={"ID":"31fecef4-0662-4ae9-870f-ca38227621e4","Type":"ContainerStarted","Data":"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"}
Sep 11 13:45:39.123443 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.123334    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" event={"ID":"31fecef4-0662-4ae9-870f-ca38227621e4","Type":"ContainerStarted","Data":"5a3c9b06c9735ebc2f3a6d0d86c42ccbd72e6d4384abdf2992b9cf76f32e89fe"}
Sep 11 13:45:39.123443 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.123360    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:39.155661 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.155586    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" podStartSLOduration=2.155569828 podStartE2EDuration="2.155569828s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:45:39.154229632 +0000 UTC m=+1570.655134105" watchObservedRunningTime="2026-09-11 13:45:39.155569828 +0000 UTC m=+1570.656474277"
Sep 11 13:45:39.185314 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:39.185251    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" podStartSLOduration=2.18522959 podStartE2EDuration="2.18522959s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:45:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:45:39.183125973 +0000 UTC m=+1570.684030447" watchObservedRunningTime="2026-09-11 13:45:39.18522959 +0000 UTC m=+1570.686134044"
Sep 11 13:45:44.414522 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:44.414489    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:45:50.129737 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:50.129700    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:45:50.130112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:50.129857    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:45:59.433569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.433525    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" podUID="385fdac2-c7d3-46ec-9374-9d6054e8830e" containerName="main" containerID="cri-o://991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203" gracePeriod=45
Sep 11 13:45:59.686520 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.686464    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:45:59.753550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753517    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.753766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753586    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.753766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753609    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.753766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753685    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.753766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753730    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-485lc\" (UniqueName: \"kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.753766 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.753749    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache\") pod \"385fdac2-c7d3-46ec-9374-9d6054e8830e\" (UID: \"385fdac2-c7d3-46ec-9374-9d6054e8830e\") "
Sep 11 13:45:59.754062 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.754009    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home" (OuterVolumeSpecName: "home") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:45:59.754162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.754130    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:45:59.754162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.754152    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache" (OuterVolumeSpecName: "model-cache") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:45:59.755851 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.755811    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm" (OuterVolumeSpecName: "dshm") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:45:59.755938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.755904    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc" (OuterVolumeSpecName: "kube-api-access-485lc") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "kube-api-access-485lc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:45:59.755938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.755905    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "385fdac2-c7d3-46ec-9374-9d6054e8830e" (UID: "385fdac2-c7d3-46ec-9374-9d6054e8830e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:45:59.854803 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854761    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:45:59.854803 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854802    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/385fdac2-c7d3-46ec-9374-9d6054e8830e-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:45:59.855041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854817    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:45:59.855041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854829    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:45:59.855041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854843    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-485lc\" (UniqueName: \"kubernetes.io/projected/385fdac2-c7d3-46ec-9374-9d6054e8830e-kube-api-access-485lc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:45:59.855041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:45:59.854857    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/385fdac2-c7d3-46ec-9374-9d6054e8830e-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:00.245570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.245538    2551 generic.go:356] "Generic (PLEG): container finished" podID="385fdac2-c7d3-46ec-9374-9d6054e8830e" containerID="991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203" exitCode=0
Sep 11 13:46:00.245796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.245604    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"
Sep 11 13:46:00.245796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.245616    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" event={"ID":"385fdac2-c7d3-46ec-9374-9d6054e8830e","Type":"ContainerDied","Data":"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"}
Sep 11 13:46:00.245796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.245695    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg" event={"ID":"385fdac2-c7d3-46ec-9374-9d6054e8830e","Type":"ContainerDied","Data":"86f0e36df21ffef7077f4d9a3d0a5399b02605f89b75f4423c2cc7c5fcbc7bce"}
Sep 11 13:46:00.245796 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.245721    2551 scope.go:117] "RemoveContainer" containerID="991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"
Sep 11 13:46:00.256934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.256917    2551 scope.go:117] "RemoveContainer" containerID="991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"
Sep 11 13:46:00.257216 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:46:00.257199    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203\": container with ID starting with 991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203 not found: ID does not exist" containerID="991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"
Sep 11 13:46:00.257284 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.257224    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203"} err="failed to get container status \"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203\": rpc error: code = NotFound desc = could not find container \"991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203\": container with ID starting with 991feeb08f7f4d1d3258bb4cd3e7f280e3160e5ca7b296732574720c992e2203 not found: ID does not exist"
Sep 11 13:46:00.283494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.283465    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:46:00.290199 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.290175    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-a-w3-3dd80c/sub-test-model-3dd80c-kserve-6b6bb5c5ff-mnhbg"]
Sep 11 13:46:00.964923 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:00.964883    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="385fdac2-c7d3-46ec-9374-9d6054e8830e" path="/var/lib/kubelet/pods/385fdac2-c7d3-46ec-9374-9d6054e8830e/volumes"
Sep 11 13:46:10.957469 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.957433    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:46:10.958058 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.958040    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="385fdac2-c7d3-46ec-9374-9d6054e8830e" containerName="main"
Sep 11 13:46:10.958126 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.958060    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="385fdac2-c7d3-46ec-9374-9d6054e8830e" containerName="main"
Sep 11 13:46:10.958184 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.958172    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="385fdac2-c7d3-46ec-9374-9d6054e8830e" containerName="main"
Sep 11 13:46:10.962708 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.962683    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:10.965419 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.965398    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-ab77a5\""
Sep 11 13:46:10.965886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.965870    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-ab77a5-dockercfg-djm4w\""
Sep 11 13:46:10.981392 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:10.981365    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:46:11.051728 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.051687    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:46:11.054870 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.054846    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.054957 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.054929    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2tz2\" (UniqueName: \"kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.055842 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.055827    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.077771 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.077741    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:46:11.155732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.155692    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2tz2\" (UniqueName: \"kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.155732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.155739    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w5tcb\" (UniqueName: \"kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.155999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.155781    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.155999 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.155892    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.156515 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.156496    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.168570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.168544    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2tz2\" (UniqueName: \"kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2\") pod \"payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.256882 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.256848    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.257075 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.256901    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w5tcb\" (UniqueName: \"kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.257658 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.257614    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.271973 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.271944    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w5tcb\" (UniqueName: \"kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb\") pod \"payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") " pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.273743 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.273721    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:11.367961 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.367933    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:11.560064 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.560036    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:46:11.562019 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:46:11.561992    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod43901419_399c_4017_9176_7886e9a8cdb0.slice/crio-fb2a5c781b46e2ceb11d478a0345400c2ae8cb76f4d24ae85f2833a86553d7f0 WatchSource:0}: Error finding container fb2a5c781b46e2ceb11d478a0345400c2ae8cb76f4d24ae85f2833a86553d7f0: Status 404 returned error can't find the container with id fb2a5c781b46e2ceb11d478a0345400c2ae8cb76f4d24ae85f2833a86553d7f0
Sep 11 13:46:11.649601 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:11.649202    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:46:11.652087 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:46:11.652056    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaaadb1b1_9800_4c63_be6e_5ac3a3f7dac8.slice/crio-29800dba54041591efe65479a45664470f4a3191aa9c8551f017c13a0e4e8fd3 WatchSource:0}: Error finding container 29800dba54041591efe65479a45664470f4a3191aa9c8551f017c13a0e4e8fd3: Status 404 returned error can't find the container with id 29800dba54041591efe65479a45664470f4a3191aa9c8551f017c13a0e4e8fd3
Sep 11 13:46:12.311675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.311578    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" event={"ID":"43901419-399c-4017-9176-7886e9a8cdb0","Type":"ContainerStarted","Data":"da21ace3c6511f41b8feb2cc535e80daae2e10e8e7bffe8a7dc0537eb5a80b69"}
Sep 11 13:46:12.311675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.311653    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" event={"ID":"43901419-399c-4017-9176-7886e9a8cdb0","Type":"ContainerStarted","Data":"fb2a5c781b46e2ceb11d478a0345400c2ae8cb76f4d24ae85f2833a86553d7f0"}
Sep 11 13:46:12.312222 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.311773    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:12.313698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.313431    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" event={"ID":"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8","Type":"ContainerStarted","Data":"ea96d1c6fa3dca16d791fd9dad2c596846481709e109d7690b82d5bc3dd32332"}
Sep 11 13:46:12.313698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.313482    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" event={"ID":"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8","Type":"ContainerStarted","Data":"29800dba54041591efe65479a45664470f4a3191aa9c8551f017c13a0e4e8fd3"}
Sep 11 13:46:12.313698 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.313509    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:12.342326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.342275    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" podStartSLOduration=2.342258323 podStartE2EDuration="2.342258323s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:46:10 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:46:12.342210208 +0000 UTC m=+1603.843114662" watchObservedRunningTime="2026-09-11 13:46:12.342258323 +0000 UTC m=+1603.843162774"
Sep 11 13:46:12.379817 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:12.379754    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" podStartSLOduration=1.379734448 podStartE2EDuration="1.379734448s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:46:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:46:12.375752605 +0000 UTC m=+1603.876657049" watchObservedRunningTime="2026-09-11 13:46:12.379734448 +0000 UTC m=+1603.880638899"
Sep 11 13:46:23.321982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:23.321914    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:46:23.321982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:23.321977    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:46:29.346095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:29.346061    2551 scope.go:117] "RemoveContainer" containerID="b7f6de540c82a61a63af9964b6f9663c79694bcb2dac50c2233d84b6941895c4"
Sep 11 13:46:29.357082 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:29.357060    2551 scope.go:117] "RemoveContainer" containerID="ca7d65834679c90aaf0f857d62c9ac89c6f26ade0ca4cd202041c4263e46a22f"
Sep 11 13:46:29.365340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:29.365319    2551 scope.go:117] "RemoveContainer" containerID="077e5bf5fd68ff8f17fa161f6380ec2d1159dfe817724c0fefa580ecc592ed98"
Sep 11 13:46:39.563891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.563859    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"]
Sep 11 13:46:39.567297 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.567279    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:39.570927 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.570896    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:46:39.571064 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.571037    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:46:39.595312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.595288    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"]
Sep 11 13:46:39.620218 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.620187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx\" (UID: \"ab837ab6-35b8-4e3e-bc45-7000e004de78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:39.721549 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.721505    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx\" (UID: \"ab837ab6-35b8-4e3e-bc45-7000e004de78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:39.722194 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.722173    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx\" (UID: \"ab837ab6-35b8-4e3e-bc45-7000e004de78\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:39.878205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:39.878125    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:40.157611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:40.157584    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"]
Sep 11 13:46:40.160289 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:46:40.160262    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab837ab6_35b8_4e3e_bc45_7000e004de78.slice/crio-5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc WatchSource:0}: Error finding container 5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc: Status 404 returned error can't find the container with id 5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc
Sep 11 13:46:40.462404 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:40.462374    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx" event={"ID":"ab837ab6-35b8-4e3e-bc45-7000e004de78","Type":"ContainerStarted","Data":"5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc"}
Sep 11 13:46:41.469311 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:41.469275    2551 generic.go:356] "Generic (PLEG): container finished" podID="ab837ab6-35b8-4e3e-bc45-7000e004de78" containerID="cc8993a2c74b38296f8ee09a0d98affcf25a7330bc0d15d5e4496c8496cfe67d" exitCode=0
Sep 11 13:46:41.469734 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:41.469364    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx" event={"ID":"ab837ab6-35b8-4e3e-bc45-7000e004de78","Type":"ContainerDied","Data":"cc8993a2c74b38296f8ee09a0d98affcf25a7330bc0d15d5e4496c8496cfe67d"}
Sep 11 13:46:42.614021 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:42.613996    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:42.648180 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:42.648149    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca\") pod \"ab837ab6-35b8-4e3e-bc45-7000e004de78\" (UID: \"ab837ab6-35b8-4e3e-bc45-7000e004de78\") "
Sep 11 13:46:42.648584 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:42.648556    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "ab837ab6-35b8-4e3e-bc45-7000e004de78" (UID: "ab837ab6-35b8-4e3e-bc45-7000e004de78"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:46:42.749685 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:42.749602    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ab837ab6-35b8-4e3e-bc45-7000e004de78-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:43.481171 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:43.481130    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx" event={"ID":"ab837ab6-35b8-4e3e-bc45-7000e004de78","Type":"ContainerDied","Data":"5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc"}
Sep 11 13:46:43.481347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:43.481181    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fff80721cd4eb27b68814aa9095577d0b53bdec73f8ab5f62c30bad0e2965cc"
Sep 11 13:46:43.481347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:43.481142    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"
Sep 11 13:46:56.308359 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.308320    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:46:56.330905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.330866    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:46:56.331127 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.331106    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" podUID="acb6968c-102c-45ed-9a95-809bccc7bb1c" containerName="payload-pre-processing" containerID="cri-o://138de6105cdebae71831bc138b636f5777603c24d92d60e1c93c101e538b3925" gracePeriod=30
Sep 11 13:46:56.549684 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.549644    2551 generic.go:356] "Generic (PLEG): container finished" podID="acb6968c-102c-45ed-9a95-809bccc7bb1c" containerID="138de6105cdebae71831bc138b636f5777603c24d92d60e1c93c101e538b3925" exitCode=0
Sep 11 13:46:56.549868 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.549703    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" event={"ID":"acb6968c-102c-45ed-9a95-809bccc7bb1c","Type":"ContainerDied","Data":"138de6105cdebae71831bc138b636f5777603c24d92d60e1c93c101e538b3925"}
Sep 11 13:46:56.550150 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.550001    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" podUID="31fecef4-0662-4ae9-870f-ca38227621e4" containerName="payload-processing" containerID="cri-o://958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9" gracePeriod=30
Sep 11 13:46:56.674915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.674889    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:46:56.775147 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.775117    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume\") pod \"acb6968c-102c-45ed-9a95-809bccc7bb1c\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") "
Sep 11 13:46:56.775312 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.775261    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9vrsk\" (UniqueName: \"kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk\") pod \"acb6968c-102c-45ed-9a95-809bccc7bb1c\" (UID: \"acb6968c-102c-45ed-9a95-809bccc7bb1c\") "
Sep 11 13:46:56.775553 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.775528    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "acb6968c-102c-45ed-9a95-809bccc7bb1c" (UID: "acb6968c-102c-45ed-9a95-809bccc7bb1c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:46:56.777549 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.777520    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk" (OuterVolumeSpecName: "kube-api-access-9vrsk") pod "acb6968c-102c-45ed-9a95-809bccc7bb1c" (UID: "acb6968c-102c-45ed-9a95-809bccc7bb1c"). InnerVolumeSpecName "kube-api-access-9vrsk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:46:56.801142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.801120    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:46:56.811739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.811715    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"]
Sep 11 13:46:56.825099 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.824994    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-65a9ed-6nhcx"]
Sep 11 13:46:56.875903 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.875875    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-9q4dl\" (UniqueName: \"kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl\") pod \"31fecef4-0662-4ae9-870f-ca38227621e4\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") "
Sep 11 13:46:56.876065 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.875972    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume\") pod \"31fecef4-0662-4ae9-870f-ca38227621e4\" (UID: \"31fecef4-0662-4ae9-870f-ca38227621e4\") "
Sep 11 13:46:56.876108 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.876088    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9vrsk\" (UniqueName: \"kubernetes.io/projected/acb6968c-102c-45ed-9a95-809bccc7bb1c-kube-api-access-9vrsk\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:56.876108 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.876099    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/acb6968c-102c-45ed-9a95-809bccc7bb1c-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:56.876352 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.876329    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "31fecef4-0662-4ae9-870f-ca38227621e4" (UID: "31fecef4-0662-4ae9-870f-ca38227621e4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:46:56.878032 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.878003    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl" (OuterVolumeSpecName: "kube-api-access-9q4dl") pod "31fecef4-0662-4ae9-870f-ca38227621e4" (UID: "31fecef4-0662-4ae9-870f-ca38227621e4"). InnerVolumeSpecName "kube-api-access-9q4dl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:46:56.967078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.967041    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ab837ab6-35b8-4e3e-bc45-7000e004de78" path="/var/lib/kubelet/pods/ab837ab6-35b8-4e3e-bc45-7000e004de78/volumes"
Sep 11 13:46:56.977087 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.977066    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9q4dl\" (UniqueName: \"kubernetes.io/projected/31fecef4-0662-4ae9-870f-ca38227621e4-kube-api-access-9q4dl\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:56.977087 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:56.977087    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/31fecef4-0662-4ae9-870f-ca38227621e4-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:46:57.556903 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.556869    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"
Sep 11 13:46:57.557369 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.556874    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt" event={"ID":"acb6968c-102c-45ed-9a95-809bccc7bb1c","Type":"ContainerDied","Data":"b81aedaf60a2fe98536e2d46b4a8f27044f5780e78969417022b0630bd22e68b"}
Sep 11 13:46:57.557369 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.557014    2551 scope.go:117] "RemoveContainer" containerID="138de6105cdebae71831bc138b636f5777603c24d92d60e1c93c101e538b3925"
Sep 11 13:46:57.558388 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.558351    2551 generic.go:356] "Generic (PLEG): container finished" podID="31fecef4-0662-4ae9-870f-ca38227621e4" containerID="958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9" exitCode=0
Sep 11 13:46:57.558471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.558402    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"
Sep 11 13:46:57.558471 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.558444    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" event={"ID":"31fecef4-0662-4ae9-870f-ca38227621e4","Type":"ContainerDied","Data":"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"}
Sep 11 13:46:57.558580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.558497    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz" event={"ID":"31fecef4-0662-4ae9-870f-ca38227621e4","Type":"ContainerDied","Data":"5a3c9b06c9735ebc2f3a6d0d86c42ccbd72e6d4384abdf2992b9cf76f32e89fe"}
Sep 11 13:46:57.565761 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.565739    2551 scope.go:117] "RemoveContainer" containerID="958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"
Sep 11 13:46:57.573861 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.573844    2551 scope.go:117] "RemoveContainer" containerID="958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"
Sep 11 13:46:57.574150 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:46:57.574129    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9\": container with ID starting with 958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9 not found: ID does not exist" containerID="958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"
Sep 11 13:46:57.574201 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.574160    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9"} err="failed to get container status \"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9\": rpc error: code = NotFound desc = could not find container \"958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9\": container with ID starting with 958a59e150c026a55403001f460ad0c164a76cc27e4435e39de935458562c3a9 not found: ID does not exist"
Sep 11 13:46:57.579881 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.579861    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:46:57.583503 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.583482    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-65a9ed-b9df58ff-z6qjz"]
Sep 11 13:46:57.608299 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.608240    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:46:57.617388 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:57.617365    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-65a9ed-74d74d4474-fgplt"]
Sep 11 13:46:58.964593 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:58.964557    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="31fecef4-0662-4ae9-870f-ca38227621e4" path="/var/lib/kubelet/pods/31fecef4-0662-4ae9-870f-ca38227621e4/volumes"
Sep 11 13:46:58.964987 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:46:58.964936    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="acb6968c-102c-45ed-9a95-809bccc7bb1c" path="/var/lib/kubelet/pods/acb6968c-102c-45ed-9a95-809bccc7bb1c/volumes"
Sep 11 13:47:06.539307 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539269    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"]
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539669    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="31fecef4-0662-4ae9-870f-ca38227621e4" containerName="payload-processing"
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539681    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="31fecef4-0662-4ae9-870f-ca38227621e4" containerName="payload-processing"
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539699    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="acb6968c-102c-45ed-9a95-809bccc7bb1c" containerName="payload-pre-processing"
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539705    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="acb6968c-102c-45ed-9a95-809bccc7bb1c" containerName="payload-pre-processing"
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539721    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ab837ab6-35b8-4e3e-bc45-7000e004de78" containerName="revoke-keys"
Sep 11 13:47:06.539732 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539728    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ab837ab6-35b8-4e3e-bc45-7000e004de78" containerName="revoke-keys"
Sep 11 13:47:06.539938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539785    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="acb6968c-102c-45ed-9a95-809bccc7bb1c" containerName="payload-pre-processing"
Sep 11 13:47:06.539938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539795    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ab837ab6-35b8-4e3e-bc45-7000e004de78" containerName="revoke-keys"
Sep 11 13:47:06.539938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.539803    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="31fecef4-0662-4ae9-870f-ca38227621e4" containerName="payload-processing"
Sep 11 13:47:06.542960 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.542943    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:06.548116 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.548092    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:47:06.548247 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.548100    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:47:06.558227 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.558201    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"]
Sep 11 13:47:06.673142 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.673105    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z\" (UID: \"12cc8577-4653-44d3-960e-b1a7bdd20a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:06.773606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.773572    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z\" (UID: \"12cc8577-4653-44d3-960e-b1a7bdd20a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:06.774271 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.774249    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z\" (UID: \"12cc8577-4653-44d3-960e-b1a7bdd20a92\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:06.855702 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:06.855607    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:07.134422 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:07.134399    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"]
Sep 11 13:47:07.136765 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:47:07.136736    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod12cc8577_4653_44d3_960e_b1a7bdd20a92.slice/crio-f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5 WatchSource:0}: Error finding container f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5: Status 404 returned error can't find the container with id f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5
Sep 11 13:47:07.611323 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:07.611287    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z" event={"ID":"12cc8577-4653-44d3-960e-b1a7bdd20a92","Type":"ContainerStarted","Data":"97962b3ae8fb90bd6e8c67d939392ae72c114168cd77e3f9d84300d7ee37c032"}
Sep 11 13:47:07.611701 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:07.611348    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z" event={"ID":"12cc8577-4653-44d3-960e-b1a7bdd20a92","Type":"ContainerStarted","Data":"f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5"}
Sep 11 13:47:07.641049 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:07.640950    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z" podStartSLOduration=1.6409334100000001 podStartE2EDuration="1.64093341s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:47:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:47:07.640350754 +0000 UTC m=+1659.141255220" watchObservedRunningTime="2026-09-11 13:47:07.64093341 +0000 UTC m=+1659.141837859"
Sep 11 13:47:08.618170 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:08.618133    2551 generic.go:356] "Generic (PLEG): container finished" podID="12cc8577-4653-44d3-960e-b1a7bdd20a92" containerID="97962b3ae8fb90bd6e8c67d939392ae72c114168cd77e3f9d84300d7ee37c032" exitCode=0
Sep 11 13:47:08.618577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:08.618224    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z" event={"ID":"12cc8577-4653-44d3-960e-b1a7bdd20a92","Type":"ContainerDied","Data":"97962b3ae8fb90bd6e8c67d939392ae72c114168cd77e3f9d84300d7ee37c032"}
Sep 11 13:47:09.764391 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:09.764363    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:09.903611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:09.903532    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca\") pod \"12cc8577-4653-44d3-960e-b1a7bdd20a92\" (UID: \"12cc8577-4653-44d3-960e-b1a7bdd20a92\") "
Sep 11 13:47:09.903942 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:09.903919    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "12cc8577-4653-44d3-960e-b1a7bdd20a92" (UID: "12cc8577-4653-44d3-960e-b1a7bdd20a92"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:10.005223 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:10.005192    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12cc8577-4653-44d3-960e-b1a7bdd20a92-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:10.634438 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:10.634404    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z" event={"ID":"12cc8577-4653-44d3-960e-b1a7bdd20a92","Type":"ContainerDied","Data":"f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5"}
Sep 11 13:47:10.634438 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:10.634422    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"
Sep 11 13:47:10.634681 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:10.634475    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f735a92369e8bb358057b0d7e2b19c9bc4763be0e87978b87736ee306b1f4fc5"
Sep 11 13:47:22.437526 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.437490    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:47:22.438168 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.437725    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" podUID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" containerName="payload-processing" containerID="cri-o://ea96d1c6fa3dca16d791fd9dad2c596846481709e109d7690b82d5bc3dd32332" gracePeriod=30
Sep 11 13:47:22.471649 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.471609    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:47:22.471854 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.471832    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" podUID="43901419-399c-4017-9176-7886e9a8cdb0" containerName="payload-pre-processing" containerID="cri-o://da21ace3c6511f41b8feb2cc535e80daae2e10e8e7bffe8a7dc0537eb5a80b69" gracePeriod=30
Sep 11 13:47:22.698068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.697988    2551 generic.go:356] "Generic (PLEG): container finished" podID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" containerID="ea96d1c6fa3dca16d791fd9dad2c596846481709e109d7690b82d5bc3dd32332" exitCode=0
Sep 11 13:47:22.698068 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.698039    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" event={"ID":"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8","Type":"ContainerDied","Data":"ea96d1c6fa3dca16d791fd9dad2c596846481709e109d7690b82d5bc3dd32332"}
Sep 11 13:47:22.700025 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.699993    2551 generic.go:356] "Generic (PLEG): container finished" podID="43901419-399c-4017-9176-7886e9a8cdb0" containerID="da21ace3c6511f41b8feb2cc535e80daae2e10e8e7bffe8a7dc0537eb5a80b69" exitCode=0
Sep 11 13:47:22.700154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.700062    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" event={"ID":"43901419-399c-4017-9176-7886e9a8cdb0","Type":"ContainerDied","Data":"da21ace3c6511f41b8feb2cc535e80daae2e10e8e7bffe8a7dc0537eb5a80b69"}
Sep 11 13:47:22.760794 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.760712    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:47:22.764316 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.764268    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:47:22.817855 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.817825    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w5tcb\" (UniqueName: \"kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb\") pod \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") "
Sep 11 13:47:22.818061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.817914    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume\") pod \"43901419-399c-4017-9176-7886e9a8cdb0\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") "
Sep 11 13:47:22.818061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.817942    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2tz2\" (UniqueName: \"kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2\") pod \"43901419-399c-4017-9176-7886e9a8cdb0\" (UID: \"43901419-399c-4017-9176-7886e9a8cdb0\") "
Sep 11 13:47:22.818213 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.818157    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume\") pod \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\" (UID: \"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8\") "
Sep 11 13:47:22.818360 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.818328    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "43901419-399c-4017-9176-7886e9a8cdb0" (UID: "43901419-399c-4017-9176-7886e9a8cdb0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:22.818447 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.818429    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/43901419-399c-4017-9176-7886e9a8cdb0-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:22.818792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.818760    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" (UID: "aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:22.820165 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.820130    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2" (OuterVolumeSpecName: "kube-api-access-c2tz2") pod "43901419-399c-4017-9176-7886e9a8cdb0" (UID: "43901419-399c-4017-9176-7886e9a8cdb0"). InnerVolumeSpecName "kube-api-access-c2tz2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:47:22.820255 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.820171    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb" (OuterVolumeSpecName: "kube-api-access-w5tcb") pod "aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" (UID: "aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8"). InnerVolumeSpecName "kube-api-access-w5tcb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:47:22.888496 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.888465    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"]
Sep 11 13:47:22.896479 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.896449    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-ab77a5-lth2z"]
Sep 11 13:47:22.918891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.918859    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:22.918891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.918885    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w5tcb\" (UniqueName: \"kubernetes.io/projected/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8-kube-api-access-w5tcb\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:22.918891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.918895    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2tz2\" (UniqueName: \"kubernetes.io/projected/43901419-399c-4017-9176-7886e9a8cdb0-kube-api-access-c2tz2\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:22.965154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:22.965083    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="12cc8577-4653-44d3-960e-b1a7bdd20a92" path="/var/lib/kubelet/pods/12cc8577-4653-44d3-960e-b1a7bdd20a92/volumes"
Sep 11 13:47:23.706324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.706291    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"
Sep 11 13:47:23.706324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.706307    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4" event={"ID":"aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8","Type":"ContainerDied","Data":"29800dba54041591efe65479a45664470f4a3191aa9c8551f017c13a0e4e8fd3"}
Sep 11 13:47:23.706874 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.706414    2551 scope.go:117] "RemoveContainer" containerID="ea96d1c6fa3dca16d791fd9dad2c596846481709e109d7690b82d5bc3dd32332"
Sep 11 13:47:23.707879 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.707831    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp" event={"ID":"43901419-399c-4017-9176-7886e9a8cdb0","Type":"ContainerDied","Data":"fb2a5c781b46e2ceb11d478a0345400c2ae8cb76f4d24ae85f2833a86553d7f0"}
Sep 11 13:47:23.707879 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.707855    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"
Sep 11 13:47:23.715133 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.715118    2551 scope.go:117] "RemoveContainer" containerID="da21ace3c6511f41b8feb2cc535e80daae2e10e8e7bffe8a7dc0537eb5a80b69"
Sep 11 13:47:23.728788 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.728766    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:47:23.730807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.730786    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-ab77a5-7cf7665cc4-rmwhp"]
Sep 11 13:47:23.745515 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.745487    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:47:23.753010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:23.752984    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-ab77a5-6c75554bdc-vfkt4"]
Sep 11 13:47:24.964619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:24.964586    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="43901419-399c-4017-9176-7886e9a8cdb0" path="/var/lib/kubelet/pods/43901419-399c-4017-9176-7886e9a8cdb0/volumes"
Sep 11 13:47:24.965024 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:24.964986    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" path="/var/lib/kubelet/pods/aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8/volumes"
Sep 11 13:47:31.522461 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.522421    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"]
Sep 11 13:47:31.523001 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.522981    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" containerName="payload-processing"
Sep 11 13:47:31.523078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523006    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" containerName="payload-processing"
Sep 11 13:47:31.523078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523043    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="43901419-399c-4017-9176-7886e9a8cdb0" containerName="payload-pre-processing"
Sep 11 13:47:31.523078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523053    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="43901419-399c-4017-9176-7886e9a8cdb0" containerName="payload-pre-processing"
Sep 11 13:47:31.523078 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523070    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="12cc8577-4653-44d3-960e-b1a7bdd20a92" containerName="revoke-keys"
Sep 11 13:47:31.523281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523079    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="12cc8577-4653-44d3-960e-b1a7bdd20a92" containerName="revoke-keys"
Sep 11 13:47:31.523281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523167    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="43901419-399c-4017-9176-7886e9a8cdb0" containerName="payload-pre-processing"
Sep 11 13:47:31.523281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523183    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="aaadb1b1-9800-4c63-be6e-5ac3a3f7dac8" containerName="payload-processing"
Sep 11 13:47:31.523281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.523193    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="12cc8577-4653-44d3-960e-b1a7bdd20a92" containerName="revoke-keys"
Sep 11 13:47:31.526160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.526139    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:31.530792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.530774    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:47:31.547404 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.547379    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:47:31.555932 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.555907    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"]
Sep 11 13:47:31.577494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.577466    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn\" (UID: \"2bb9d3c2-4b61-49c0-9c0d-81a781c19468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:31.678284 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.678250    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn\" (UID: \"2bb9d3c2-4b61-49c0-9c0d-81a781c19468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:31.678875 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.678854    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn\" (UID: \"2bb9d3c2-4b61-49c0-9c0d-81a781c19468\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:31.836167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:31.836063    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:32.124324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:32.124298    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"]
Sep 11 13:47:32.126781 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:47:32.126755    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2bb9d3c2_4b61_49c0_9c0d_81a781c19468.slice/crio-3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d WatchSource:0}: Error finding container 3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d: Status 404 returned error can't find the container with id 3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d
Sep 11 13:47:32.762175 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:32.762143    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn" event={"ID":"2bb9d3c2-4b61-49c0-9c0d-81a781c19468","Type":"ContainerStarted","Data":"27869d90b3d37f4bf1ab8229e1c5d4bc85f58a2eeaf1c2eda0d54d3ee5f48f24"}
Sep 11 13:47:32.762590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:32.762201    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn" event={"ID":"2bb9d3c2-4b61-49c0-9c0d-81a781c19468","Type":"ContainerStarted","Data":"3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d"}
Sep 11 13:47:32.808403 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:32.808350    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn" podStartSLOduration=1.8083315070000001 podStartE2EDuration="1.808331507s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:47:32.795981915 +0000 UTC m=+1684.296886363" watchObservedRunningTime="2026-09-11 13:47:32.808331507 +0000 UTC m=+1684.309235956"
Sep 11 13:47:33.768453 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:33.768416    2551 generic.go:356] "Generic (PLEG): container finished" podID="2bb9d3c2-4b61-49c0-9c0d-81a781c19468" containerID="27869d90b3d37f4bf1ab8229e1c5d4bc85f58a2eeaf1c2eda0d54d3ee5f48f24" exitCode=0
Sep 11 13:47:33.768854 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:33.768496    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn" event={"ID":"2bb9d3c2-4b61-49c0-9c0d-81a781c19468","Type":"ContainerDied","Data":"27869d90b3d37f4bf1ab8229e1c5d4bc85f58a2eeaf1c2eda0d54d3ee5f48f24"}
Sep 11 13:47:34.908905 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:34.908881    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:35.005165 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.005129    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca\") pod \"2bb9d3c2-4b61-49c0-9c0d-81a781c19468\" (UID: \"2bb9d3c2-4b61-49c0-9c0d-81a781c19468\") "
Sep 11 13:47:35.005576 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.005546    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "2bb9d3c2-4b61-49c0-9c0d-81a781c19468" (UID: "2bb9d3c2-4b61-49c0-9c0d-81a781c19468"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:35.106697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.106602    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2bb9d3c2-4b61-49c0-9c0d-81a781c19468-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:35.780911 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.780876    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn" event={"ID":"2bb9d3c2-4b61-49c0-9c0d-81a781c19468","Type":"ContainerDied","Data":"3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d"}
Sep 11 13:47:35.781082 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.780917    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"
Sep 11 13:47:35.781082 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:35.780933    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3763729e11bc92c262d2193756d0850f5a9919f66b5c586c4fa854ea8ac5b87d"
Sep 11 13:47:42.370518 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.370484    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"]
Sep 11 13:47:42.383527 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.383495    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w2-7bfa93-29818905-mw6zq"]
Sep 11 13:47:42.430180 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.430149    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:47:42.430444 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.430402    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" podUID="774ccd65-5059-42c3-92be-1b6472cfaa3c" containerName="payload-processing" containerID="cri-o://8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8" gracePeriod=30
Sep 11 13:47:42.460406 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.460375    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:47:42.460618 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.460572    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" podUID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" containerName="payload-pre-processing" containerID="cri-o://96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd" gracePeriod=30
Sep 11 13:47:42.734363 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.734338    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:47:42.737198 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.737180    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:47:42.776834 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.776802    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume\") pod \"774ccd65-5059-42c3-92be-1b6472cfaa3c\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") "
Sep 11 13:47:42.777015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.776847    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sgbr5\" (UniqueName: \"kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5\") pod \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") "
Sep 11 13:47:42.777015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.776900    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-pdg27\" (UniqueName: \"kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27\") pod \"774ccd65-5059-42c3-92be-1b6472cfaa3c\" (UID: \"774ccd65-5059-42c3-92be-1b6472cfaa3c\") "
Sep 11 13:47:42.777015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.776992    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume\") pod \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\" (UID: \"fa61f7dd-b7d6-452f-aa62-2989e19ead01\") "
Sep 11 13:47:42.777437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.777407    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "fa61f7dd-b7d6-452f-aa62-2989e19ead01" (UID: "fa61f7dd-b7d6-452f-aa62-2989e19ead01"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:42.777559 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.777514    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "774ccd65-5059-42c3-92be-1b6472cfaa3c" (UID: "774ccd65-5059-42c3-92be-1b6472cfaa3c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:42.779315 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.779291    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27" (OuterVolumeSpecName: "kube-api-access-pdg27") pod "774ccd65-5059-42c3-92be-1b6472cfaa3c" (UID: "774ccd65-5059-42c3-92be-1b6472cfaa3c"). InnerVolumeSpecName "kube-api-access-pdg27". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:47:42.779401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.779309    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5" (OuterVolumeSpecName: "kube-api-access-sgbr5") pod "fa61f7dd-b7d6-452f-aa62-2989e19ead01" (UID: "fa61f7dd-b7d6-452f-aa62-2989e19ead01"). InnerVolumeSpecName "kube-api-access-sgbr5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:47:42.818245 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.818212    2551 generic.go:356] "Generic (PLEG): container finished" podID="774ccd65-5059-42c3-92be-1b6472cfaa3c" containerID="8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8" exitCode=0
Sep 11 13:47:42.818418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.818260    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"
Sep 11 13:47:42.818418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.818308    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" event={"ID":"774ccd65-5059-42c3-92be-1b6472cfaa3c","Type":"ContainerDied","Data":"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"}
Sep 11 13:47:42.818418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.818372    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2" event={"ID":"774ccd65-5059-42c3-92be-1b6472cfaa3c","Type":"ContainerDied","Data":"a256c18c631feede0d2b28ab388d365afd456c6f575def1479dd9bcbe8c5d8da"}
Sep 11 13:47:42.818418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.818406    2551 scope.go:117] "RemoveContainer" containerID="8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"
Sep 11 13:47:42.820112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.820061    2551 generic.go:356] "Generic (PLEG): container finished" podID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" containerID="96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd" exitCode=0
Sep 11 13:47:42.820176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.820147    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" event={"ID":"fa61f7dd-b7d6-452f-aa62-2989e19ead01","Type":"ContainerDied","Data":"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"}
Sep 11 13:47:42.820243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.820190    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq" event={"ID":"fa61f7dd-b7d6-452f-aa62-2989e19ead01","Type":"ContainerDied","Data":"84290f64ea43f8d8b5ecca55c4da5a2bbcb361afc816a7439b5473e0e58bf409"}
Sep 11 13:47:42.820309 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.820257    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"
Sep 11 13:47:42.828711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.828691    2551 scope.go:117] "RemoveContainer" containerID="8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"
Sep 11 13:47:42.828982 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:47:42.828964    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8\": container with ID starting with 8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8 not found: ID does not exist" containerID="8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"
Sep 11 13:47:42.829048 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.828989    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8"} err="failed to get container status \"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8\": rpc error: code = NotFound desc = could not find container \"8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8\": container with ID starting with 8e2b17f897a674931e158c37aa156431c7de6ed7549365f2f1add872d70a5ac8 not found: ID does not exist"
Sep 11 13:47:42.829048 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.829009    2551 scope.go:117] "RemoveContainer" containerID="96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"
Sep 11 13:47:42.839338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.839318    2551 scope.go:117] "RemoveContainer" containerID="96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"
Sep 11 13:47:42.839661 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:47:42.839616    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd\": container with ID starting with 96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd not found: ID does not exist" containerID="96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"
Sep 11 13:47:42.839730 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.839667    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd"} err="failed to get container status \"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd\": rpc error: code = NotFound desc = could not find container \"96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd\": container with ID starting with 96cc10a35bcd28f95fc78c26174549038ebeb6154123b2daabab974e825841cd not found: ID does not exist"
Sep 11 13:47:42.849801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.849775    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:47:42.859250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.859225    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w2-7bfa93-84967d7969-htcc2"]
Sep 11 13:47:42.865044 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.865025    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"]
Sep 11 13:47:42.868521 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.868502    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w2-7bfa93-mzggn"]
Sep 11 13:47:42.878229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.878209    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/774ccd65-5059-42c3-92be-1b6472cfaa3c-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:42.878229 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.878229    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sgbr5\" (UniqueName: \"kubernetes.io/projected/fa61f7dd-b7d6-452f-aa62-2989e19ead01-kube-api-access-sgbr5\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:42.878350 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.878239    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pdg27\" (UniqueName: \"kubernetes.io/projected/774ccd65-5059-42c3-92be-1b6472cfaa3c-kube-api-access-pdg27\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:42.878350 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.878248    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/fa61f7dd-b7d6-452f-aa62-2989e19ead01-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:42.880769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.880750    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:47:42.884368 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.884349    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w2-7bfa93-7f8cd7c646-4zcsq"]
Sep 11 13:47:42.965265 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.965182    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2bb9d3c2-4b61-49c0-9c0d-81a781c19468" path="/var/lib/kubelet/pods/2bb9d3c2-4b61-49c0-9c0d-81a781c19468/volumes"
Sep 11 13:47:42.965552 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.965538    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="774ccd65-5059-42c3-92be-1b6472cfaa3c" path="/var/lib/kubelet/pods/774ccd65-5059-42c3-92be-1b6472cfaa3c/volumes"
Sep 11 13:47:42.965795 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.965783    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="bb1add6b-a09a-45e6-a11b-90c50366e5dd" path="/var/lib/kubelet/pods/bb1add6b-a09a-45e6-a11b-90c50366e5dd/volumes"
Sep 11 13:47:42.966063 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:42.966050    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" path="/var/lib/kubelet/pods/fa61f7dd-b7d6-452f-aa62-2989e19ead01/volumes"
Sep 11 13:47:51.455119 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455025    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"]
Sep 11 13:47:51.455586 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455538    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="2bb9d3c2-4b61-49c0-9c0d-81a781c19468" containerName="revoke-keys"
Sep 11 13:47:51.455586 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455554    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="2bb9d3c2-4b61-49c0-9c0d-81a781c19468" containerName="revoke-keys"
Sep 11 13:47:51.455586 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455579    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" containerName="payload-pre-processing"
Sep 11 13:47:51.455586 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455589    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" containerName="payload-pre-processing"
Sep 11 13:47:51.455785 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455604    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="774ccd65-5059-42c3-92be-1b6472cfaa3c" containerName="payload-processing"
Sep 11 13:47:51.455785 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.455614    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="774ccd65-5059-42c3-92be-1b6472cfaa3c" containerName="payload-processing"
Sep 11 13:47:51.462141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.456160    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="fa61f7dd-b7d6-452f-aa62-2989e19ead01" containerName="payload-pre-processing"
Sep 11 13:47:51.462141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.456211    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="774ccd65-5059-42c3-92be-1b6472cfaa3c" containerName="payload-processing"
Sep 11 13:47:51.462141 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.456235    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="2bb9d3c2-4b61-49c0-9c0d-81a781c19468" containerName="revoke-keys"
Sep 11 13:47:51.464959 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.464930    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:51.470279 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.470252    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:47:51.470413 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.470254    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:47:51.473937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.473907    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"]
Sep 11 13:47:51.554519 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.554475    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g\" (UID: \"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:51.655449 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.655413    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g\" (UID: \"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:51.656072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.656053    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g\" (UID: \"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:51.777325 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:51.777290    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:52.060371 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:52.060334    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"]
Sep 11 13:47:52.062792 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:47:52.062757    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod097e0d9e_244f_4291_8dbb_d15f9c1c8c0d.slice/crio-551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c WatchSource:0}: Error finding container 551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c: Status 404 returned error can't find the container with id 551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c
Sep 11 13:47:52.876777 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:52.876750    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g" event={"ID":"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d","Type":"ContainerStarted","Data":"c6e887735e044d0b8ea75d68429dc8c0ff960346f5101a6e02d8aeb70690925e"}
Sep 11 13:47:52.877112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:52.876805    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g" event={"ID":"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d","Type":"ContainerStarted","Data":"551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c"}
Sep 11 13:47:53.883267 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:53.883231    2551 generic.go:356] "Generic (PLEG): container finished" podID="097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" containerID="c6e887735e044d0b8ea75d68429dc8c0ff960346f5101a6e02d8aeb70690925e" exitCode=0
Sep 11 13:47:53.883267 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:53.883272    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g" event={"ID":"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d","Type":"ContainerDied","Data":"c6e887735e044d0b8ea75d68429dc8c0ff960346f5101a6e02d8aeb70690925e"}
Sep 11 13:47:54.017034 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.017005    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:54.076013 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.075982    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca\") pod \"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d\" (UID: \"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d\") "
Sep 11 13:47:54.076326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.076303    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" (UID: "097e0d9e-244f-4291-8dbb-d15f9c1c8c0d"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:47:54.177205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.177120    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:47:54.889192 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.889163    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"
Sep 11 13:47:54.889192 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.889177    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g" event={"ID":"097e0d9e-244f-4291-8dbb-d15f9c1c8c0d","Type":"ContainerDied","Data":"551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c"}
Sep 11 13:47:54.889658 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:54.889226    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="551595213ac291fb2a932db7ce76b4a17699d7d3b70df6723031716b4242655c"
Sep 11 13:47:59.047394 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.047355    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:47:59.047998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.047978    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" containerName="revoke-keys"
Sep 11 13:47:59.048081 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.048003    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" containerName="revoke-keys"
Sep 11 13:47:59.048124 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.048093    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" containerName="revoke-keys"
Sep 11 13:47:59.051694 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.051676    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.055728 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.055706    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-a-4920fa\""
Sep 11 13:47:59.055861 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.055759    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-a-4920fa-dockercfg-pg8bs\""
Sep 11 13:47:59.068752 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.068726    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:47:59.122650 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.122598    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.122854 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.122747    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tn992\" (UniqueName: \"kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.126762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.126739    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:47:59.130117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.130101    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.141982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.141955    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:47:59.223617 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.223587    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tn992\" (UniqueName: \"kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.223786 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.223673    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kx2qw\" (UniqueName: \"kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.223786 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.223700    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.223786 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.223718    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.224366 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.224348    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.245365 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.245341    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-tn992\" (UniqueName: \"kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992\") pod \"payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.324877 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.324796    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.325024 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.324903    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kx2qw\" (UniqueName: \"kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.325506 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.325487    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.335168 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.335140    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-kx2qw\" (UniqueName: \"kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw\") pod \"payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") " pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.363604 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.363573    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.440697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.440662    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.657025 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.656994    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:47:59.658811 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:47:59.658781    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod16743710_046e_4704_99b9_8220f32ca42c.slice/crio-bebc3243a682cbd82c624b2384999b3ee12b1a49735bcf501cf103a591359844 WatchSource:0}: Error finding container bebc3243a682cbd82c624b2384999b3ee12b1a49735bcf501cf103a591359844: Status 404 returned error can't find the container with id bebc3243a682cbd82c624b2384999b3ee12b1a49735bcf501cf103a591359844
Sep 11 13:47:59.724870 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.724844    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:47:59.728487 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:47:59.728463    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06d50f02_87dc_4a80_80b7_9e35a20ff245.slice/crio-662267bf84f7a96ea121d025c98b7c74d860adc55a64a8b1aa868637b41493b0 WatchSource:0}: Error finding container 662267bf84f7a96ea121d025c98b7c74d860adc55a64a8b1aa868637b41493b0: Status 404 returned error can't find the container with id 662267bf84f7a96ea121d025c98b7c74d860adc55a64a8b1aa868637b41493b0
Sep 11 13:47:59.917775 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.917670    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" event={"ID":"16743710-046e-4704-99b9-8220f32ca42c","Type":"ContainerStarted","Data":"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"}
Sep 11 13:47:59.917775 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.917738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" event={"ID":"16743710-046e-4704-99b9-8220f32ca42c","Type":"ContainerStarted","Data":"bebc3243a682cbd82c624b2384999b3ee12b1a49735bcf501cf103a591359844"}
Sep 11 13:47:59.917995 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.917876    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:47:59.919401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.919375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" event={"ID":"06d50f02-87dc-4a80-80b7-9e35a20ff245","Type":"ContainerStarted","Data":"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"}
Sep 11 13:47:59.919520 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.919424    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:47:59.919520 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.919447    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" event={"ID":"06d50f02-87dc-4a80-80b7-9e35a20ff245","Type":"ContainerStarted","Data":"662267bf84f7a96ea121d025c98b7c74d860adc55a64a8b1aa868637b41493b0"}
Sep 11 13:47:59.962494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.962413    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" podStartSLOduration=0.962393139 podStartE2EDuration="962.393139ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:47:59.962072389 +0000 UTC m=+1711.462976851" watchObservedRunningTime="2026-09-11 13:47:59.962393139 +0000 UTC m=+1711.463297592"
Sep 11 13:47:59.964054 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:47:59.964013    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" podStartSLOduration=0.963997824 podStartE2EDuration="963.997824ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:47:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:47:59.940801975 +0000 UTC m=+1711.441706442" watchObservedRunningTime="2026-09-11 13:47:59.963997824 +0000 UTC m=+1711.464902277"
Sep 11 13:48:01.596510 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.596474    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"]
Sep 11 13:48:01.599156 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.599129    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w2-32b673-29818905-pg9f9"]
Sep 11 13:48:01.658247 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.658206    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:48:01.658601 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.658564    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" podUID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" containerName="payload-processing" containerID="cri-o://618d2930c4ede3e02580c3c78e07473bb894a2e32d78dd124da622db90790a8e" gracePeriod=30
Sep 11 13:48:01.666086 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.666063    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:48:01.666273 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.666252    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" podUID="402d92e9-acd7-4aa0-b168-f18bd09fa187" containerName="payload-pre-processing" containerID="cri-o://1282efb033e7440ee27cfafd891336089e92c063220d80931f0345291ac01a46" gracePeriod=30
Sep 11 13:48:01.933008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.932975    2551 generic.go:356] "Generic (PLEG): container finished" podID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" containerID="618d2930c4ede3e02580c3c78e07473bb894a2e32d78dd124da622db90790a8e" exitCode=0
Sep 11 13:48:01.933154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.933040    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" event={"ID":"cf5e7252-1c6e-437f-9378-69c6cf87fe30","Type":"ContainerDied","Data":"618d2930c4ede3e02580c3c78e07473bb894a2e32d78dd124da622db90790a8e"}
Sep 11 13:48:01.934731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.934611    2551 generic.go:356] "Generic (PLEG): container finished" podID="402d92e9-acd7-4aa0-b168-f18bd09fa187" containerID="1282efb033e7440ee27cfafd891336089e92c063220d80931f0345291ac01a46" exitCode=0
Sep 11 13:48:01.934731 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.934675    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" event={"ID":"402d92e9-acd7-4aa0-b168-f18bd09fa187","Type":"ContainerDied","Data":"1282efb033e7440ee27cfafd891336089e92c063220d80931f0345291ac01a46"}
Sep 11 13:48:01.943001 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.942985    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:48:01.954794 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.954768    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"]
Sep 11 13:48:01.959084 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.959060    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:48:01.961418 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:01.961397    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w2-32b673-dt54g"]
Sep 11 13:48:02.050657 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.050593    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-22cl2\" (UniqueName: \"kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2\") pod \"402d92e9-acd7-4aa0-b168-f18bd09fa187\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") "
Sep 11 13:48:02.050838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.050676    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume\") pod \"402d92e9-acd7-4aa0-b168-f18bd09fa187\" (UID: \"402d92e9-acd7-4aa0-b168-f18bd09fa187\") "
Sep 11 13:48:02.050838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.050721    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume\") pod \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") "
Sep 11 13:48:02.050838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.050750    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-f5nb5\" (UniqueName: \"kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5\") pod \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\" (UID: \"cf5e7252-1c6e-437f-9378-69c6cf87fe30\") "
Sep 11 13:48:02.051157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.051114    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "cf5e7252-1c6e-437f-9378-69c6cf87fe30" (UID: "cf5e7252-1c6e-437f-9378-69c6cf87fe30"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:48:02.051157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.051125    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "402d92e9-acd7-4aa0-b168-f18bd09fa187" (UID: "402d92e9-acd7-4aa0-b168-f18bd09fa187"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:48:02.052915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.052879    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2" (OuterVolumeSpecName: "kube-api-access-22cl2") pod "402d92e9-acd7-4aa0-b168-f18bd09fa187" (UID: "402d92e9-acd7-4aa0-b168-f18bd09fa187"). InnerVolumeSpecName "kube-api-access-22cl2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:48:02.053326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.053302    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5" (OuterVolumeSpecName: "kube-api-access-f5nb5") pod "cf5e7252-1c6e-437f-9378-69c6cf87fe30" (UID: "cf5e7252-1c6e-437f-9378-69c6cf87fe30"). InnerVolumeSpecName "kube-api-access-f5nb5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:48:02.152358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.152274    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-22cl2\" (UniqueName: \"kubernetes.io/projected/402d92e9-acd7-4aa0-b168-f18bd09fa187-kube-api-access-22cl2\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:48:02.152358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.152317    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/402d92e9-acd7-4aa0-b168-f18bd09fa187-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:48:02.152358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.152332    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf5e7252-1c6e-437f-9378-69c6cf87fe30-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:48:02.152358 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.152345    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nb5\" (UniqueName: \"kubernetes.io/projected/cf5e7252-1c6e-437f-9378-69c6cf87fe30-kube-api-access-f5nb5\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:48:02.941611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.941512    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"
Sep 11 13:48:02.941611 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.941568    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt" event={"ID":"cf5e7252-1c6e-437f-9378-69c6cf87fe30","Type":"ContainerDied","Data":"35d30bb45b62e90fbdba6903b533e8d3a50685ec6551aac2bf912d0410bc1c77"}
Sep 11 13:48:02.942164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.941667    2551 scope.go:117] "RemoveContainer" containerID="618d2930c4ede3e02580c3c78e07473bb894a2e32d78dd124da622db90790a8e"
Sep 11 13:48:02.943117 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.943098    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4" event={"ID":"402d92e9-acd7-4aa0-b168-f18bd09fa187","Type":"ContainerDied","Data":"e744fabcd380dd86245681da11aec2da24fc8cb9d3b154fdc06ec3654fae5ab8"}
Sep 11 13:48:02.943176 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.943133    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"
Sep 11 13:48:02.952534 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.952509    2551 scope.go:117] "RemoveContainer" containerID="1282efb033e7440ee27cfafd891336089e92c063220d80931f0345291ac01a46"
Sep 11 13:48:02.964830 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.964802    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="097e0d9e-244f-4291-8dbb-d15f9c1c8c0d" path="/var/lib/kubelet/pods/097e0d9e-244f-4291-8dbb-d15f9c1c8c0d/volumes"
Sep 11 13:48:02.965076 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.965059    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4ff2946f-3d89-40c4-aaf5-f5b8e4791006" path="/var/lib/kubelet/pods/4ff2946f-3d89-40c4-aaf5-f5b8e4791006/volumes"
Sep 11 13:48:02.968400 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.968377    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:48:02.973345 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.973323    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w2-32b673-5b998bc64f-7hxvt"]
Sep 11 13:48:02.992977 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.992948    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:48:02.997038 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:02.997009    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w2-32b673-785bb5855d-mxjl4"]
Sep 11 13:48:04.968243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:04.968212    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="402d92e9-acd7-4aa0-b168-f18bd09fa187" path="/var/lib/kubelet/pods/402d92e9-acd7-4aa0-b168-f18bd09fa187/volumes"
Sep 11 13:48:04.968642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:04.968547    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" path="/var/lib/kubelet/pods/cf5e7252-1c6e-437f-9378-69c6cf87fe30/volumes"
Sep 11 13:48:10.926677 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:10.926622    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:48:10.927061 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:10.926773    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.694447    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695253    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="402d92e9-acd7-4aa0-b168-f18bd09fa187" containerName="payload-pre-processing"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695274    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="402d92e9-acd7-4aa0-b168-f18bd09fa187" containerName="payload-pre-processing"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695322    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" containerName="payload-processing"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695330    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" containerName="payload-processing"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695469    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="402d92e9-acd7-4aa0-b168-f18bd09fa187" containerName="payload-pre-processing"
Sep 11 13:48:35.697097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.695490    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf5e7252-1c6e-437f-9378-69c6cf87fe30" containerName="payload-processing"
Sep 11 13:48:35.699500 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.699473    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.707208 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.707191    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-b-9efb13\""
Sep 11 13:48:35.723569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.723542    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n7xln\" (UniqueName: \"kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.723699 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.723581    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.824992 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.824958    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n7xln\" (UniqueName: \"kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.825172 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.825007    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.825781 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.825761    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.866710 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.866679    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-n7xln\" (UniqueName: \"kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln\") pod \"payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:35.901555 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:35.901516    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:48:36.009721 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.009682    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:36.260185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.260108    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:48:36.264692 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.264667    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.279790 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.279757    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-b-9efb13-dockercfg-gtmsc\""
Sep 11 13:48:36.305881 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.305851    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:48:36.330109 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.330080    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w4gch\" (UniqueName: \"kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.330277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.330203    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.372109 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.372083    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:48:36.373809 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:48:36.373780    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b7dc5fe_04c6_4f88_ab11_10a46a467c4b.slice/crio-a382f6ed47de5c1e6570b5b31ad9cd1110fba509b88d9240340cb813c3416d8c WatchSource:0}: Error finding container a382f6ed47de5c1e6570b5b31ad9cd1110fba509b88d9240340cb813c3416d8c: Status 404 returned error can't find the container with id a382f6ed47de5c1e6570b5b31ad9cd1110fba509b88d9240340cb813c3416d8c
Sep 11 13:48:36.431519 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.431494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.431654 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.431539    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w4gch\" (UniqueName: \"kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.432201 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.432181    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.465314 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.465283    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-w4gch\" (UniqueName: \"kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch\") pod \"payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") " pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.575571 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.575483    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:36.869976 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:36.869929    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:48:36.872998 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:48:36.872969    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac2a335b_f96d_482e_ab74_d6f371a50021.slice/crio-6e8485c7c1095dd3a2fa2192a94ec41b0bd8124fe5342c543c4e2e7c9927bd87 WatchSource:0}: Error finding container 6e8485c7c1095dd3a2fa2192a94ec41b0bd8124fe5342c543c4e2e7c9927bd87: Status 404 returned error can't find the container with id 6e8485c7c1095dd3a2fa2192a94ec41b0bd8124fe5342c543c4e2e7c9927bd87
Sep 11 13:48:37.128164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.128060    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" event={"ID":"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b","Type":"ContainerStarted","Data":"0c814495c3caede130f817aa6411f65bb2fb67f9fecdf109155c415e91e24675"}
Sep 11 13:48:37.128164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.128135    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:37.128164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.128163    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" event={"ID":"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b","Type":"ContainerStarted","Data":"a382f6ed47de5c1e6570b5b31ad9cd1110fba509b88d9240340cb813c3416d8c"}
Sep 11 13:48:37.129582 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.129558    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" event={"ID":"ac2a335b-f96d-482e-ab74-d6f371a50021","Type":"ContainerStarted","Data":"57aa33943f094ac497efdf751308633525cdae90c07693a66d4aea5b2129ea3e"}
Sep 11 13:48:37.129722 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.129600    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" event={"ID":"ac2a335b-f96d-482e-ab74-d6f371a50021","Type":"ContainerStarted","Data":"6e8485c7c1095dd3a2fa2192a94ec41b0bd8124fe5342c543c4e2e7c9927bd87"}
Sep 11 13:48:37.129722 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.129673    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:48:37.171482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.171427    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" podStartSLOduration=2.171408232 podStartE2EDuration="2.171408232s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:48:37.171155146 +0000 UTC m=+1748.672059617" watchObservedRunningTime="2026-09-11 13:48:37.171408232 +0000 UTC m=+1748.672312680"
Sep 11 13:48:37.215524 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:37.215473    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" podStartSLOduration=1.215456645 podStartE2EDuration="1.215456645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:48:37.212426288 +0000 UTC m=+1748.713330738" watchObservedRunningTime="2026-09-11 13:48:37.215456645 +0000 UTC m=+1748.716361093"
Sep 11 13:48:48.137079 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:48.137044    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:48:48.137590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:48:48.137214    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:49:04.732091 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:04.732058    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"
Sep 11 13:49:04.871577 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:04.871546    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf_ac2a335b-f96d-482e-ab74-d6f371a50021/payload-processing/0.log"
Sep 11 13:49:29.440934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:29.440902    2551 scope.go:117] "RemoveContainer" containerID="2b7285ed8fbeb76d36c020b956cc4ba4a6befea17edc5970a502a31807b2488e"
Sep 11 13:49:31.323240 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:31.323210    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8_06d50f02-87dc-4a80-80b7-9e35a20ff245/payload-processing/0.log"
Sep 11 13:49:31.468390 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:31.468361    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"
Sep 11 13:49:31.624490 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:31.624414    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf_ac2a335b-f96d-482e-ab74-d6f371a50021/payload-processing/0.log"
Sep 11 13:49:41.336782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.336745    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:49:41.340333 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.340316    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.343346 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.343322    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ipp-cleanup-2e3865-dockercfg-4qjp5\""
Sep 11 13:49:41.343512 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.343494    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ipp-cleanup-2e3865\""
Sep 11 13:49:41.351610 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.351581    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:49:41.405374 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.405329    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.405545 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.405377    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fw2kc\" (UniqueName: \"kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.407424 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.407403    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:49:41.411006 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.410981    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.424470 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.424445    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:49:41.506003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.505965    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.506182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.506071    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrvh6\" (UniqueName: \"kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.506182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.506142    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.506182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.506167    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fw2kc\" (UniqueName: \"kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.506826 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.506808    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.517679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.517648    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fw2kc\" (UniqueName: \"kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc\") pod \"payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") " pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.607570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.607488    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.607570 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.607554    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrvh6\" (UniqueName: \"kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.608199 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.608178    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.618543 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.618517    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrvh6\" (UniqueName: \"kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6\") pod \"payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") " pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.653426 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.653385    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:41.723436 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.723408    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:41.930934 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:41.930906    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:49:41.931812 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:49:41.931784    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2534b7d_18bb_4b28_a989_ffb24f1b5936.slice/crio-f15e8b93ceb22b2bdc5f3e3f7384a58c9a841616019a9ad62116fcd029b2e896 WatchSource:0}: Error finding container f15e8b93ceb22b2bdc5f3e3f7384a58c9a841616019a9ad62116fcd029b2e896: Status 404 returned error can't find the container with id f15e8b93ceb22b2bdc5f3e3f7384a58c9a841616019a9ad62116fcd029b2e896
Sep 11 13:49:42.006239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.006217    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:49:42.009236 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:49:42.009208    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod279acda3_5f66_48af_94dd_d6f3664f407b.slice/crio-74ae123e42185ed8287c57dd51148c3d29758122c67b76f37ee923ee54a88ed2 WatchSource:0}: Error finding container 74ae123e42185ed8287c57dd51148c3d29758122c67b76f37ee923ee54a88ed2: Status 404 returned error can't find the container with id 74ae123e42185ed8287c57dd51148c3d29758122c67b76f37ee923ee54a88ed2
Sep 11 13:49:42.470566 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.470529    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" event={"ID":"c2534b7d-18bb-4b28-a989-ffb24f1b5936","Type":"ContainerStarted","Data":"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"}
Sep 11 13:49:42.471056 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.470614    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" event={"ID":"c2534b7d-18bb-4b28-a989-ffb24f1b5936","Type":"ContainerStarted","Data":"f15e8b93ceb22b2bdc5f3e3f7384a58c9a841616019a9ad62116fcd029b2e896"}
Sep 11 13:49:42.471056 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.470658    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:42.472278 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.472242    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" event={"ID":"279acda3-5f66-48af-94dd-d6f3664f407b","Type":"ContainerStarted","Data":"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"}
Sep 11 13:49:42.472413 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.472297    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" event={"ID":"279acda3-5f66-48af-94dd-d6f3664f407b","Type":"ContainerStarted","Data":"74ae123e42185ed8287c57dd51148c3d29758122c67b76f37ee923ee54a88ed2"}
Sep 11 13:49:42.472413 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.472325    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:49:42.490835 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.490783    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" podStartSLOduration=1.490764129 podStartE2EDuration="1.490764129s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:42.490702435 +0000 UTC m=+1813.991606905" watchObservedRunningTime="2026-09-11 13:49:42.490764129 +0000 UTC m=+1813.991668579"
Sep 11 13:49:42.520286 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:42.520232    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" podStartSLOduration=1.520214771 podStartE2EDuration="1.520214771s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:49:42.518786657 +0000 UTC m=+1814.019691110" watchObservedRunningTime="2026-09-11 13:49:42.520214771 +0000 UTC m=+1814.021119220"
Sep 11 13:49:53.479487 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:53.479454    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:49:53.480011 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:49:53.479754    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:50:07.788939 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.788896    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"]
Sep 11 13:50:07.792452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.792428    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:07.795889 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.795867    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:50:07.796778 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.796758    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:50:07.805735 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.805707    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"]
Sep 11 13:50:07.945819 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:07.945776    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j\" (UID: \"d12262da-8594-4f54-8437-e08c5ac83db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:08.046886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.046785    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j\" (UID: \"d12262da-8594-4f54-8437-e08c5ac83db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:08.047438 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.047414    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j\" (UID: \"d12262da-8594-4f54-8437-e08c5ac83db5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:08.103378 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.103350    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:08.390037 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.390007    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"]
Sep 11 13:50:08.391114 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:50:08.391084    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd12262da_8594_4f54_8437_e08c5ac83db5.slice/crio-98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0 WatchSource:0}: Error finding container 98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0: Status 404 returned error can't find the container with id 98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0
Sep 11 13:50:08.615998 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.615962    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j" event={"ID":"d12262da-8594-4f54-8437-e08c5ac83db5","Type":"ContainerStarted","Data":"1ee885a0fadd0be8a3dc694a8464975c03e2d8189fece49766e4abc26eaece27"}
Sep 11 13:50:08.616160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.616016    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j" event={"ID":"d12262da-8594-4f54-8437-e08c5ac83db5","Type":"ContainerStarted","Data":"98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0"}
Sep 11 13:50:08.636051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:08.635998    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j" podStartSLOduration=1.635981286 podStartE2EDuration="1.635981286s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:50:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:50:08.634507894 +0000 UTC m=+1840.135412366" watchObservedRunningTime="2026-09-11 13:50:08.635981286 +0000 UTC m=+1840.136885734"
Sep 11 13:50:09.623028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:09.622991    2551 generic.go:356] "Generic (PLEG): container finished" podID="d12262da-8594-4f54-8437-e08c5ac83db5" containerID="1ee885a0fadd0be8a3dc694a8464975c03e2d8189fece49766e4abc26eaece27" exitCode=0
Sep 11 13:50:09.623414 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:09.623038    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j" event={"ID":"d12262da-8594-4f54-8437-e08c5ac83db5","Type":"ContainerDied","Data":"1ee885a0fadd0be8a3dc694a8464975c03e2d8189fece49766e4abc26eaece27"}
Sep 11 13:50:10.760616 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:10.760588    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:10.770696 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:10.770671    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca\") pod \"d12262da-8594-4f54-8437-e08c5ac83db5\" (UID: \"d12262da-8594-4f54-8437-e08c5ac83db5\") "
Sep 11 13:50:10.770993 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:10.770970    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "d12262da-8594-4f54-8437-e08c5ac83db5" (UID: "d12262da-8594-4f54-8437-e08c5ac83db5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:10.871233 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:10.871188    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d12262da-8594-4f54-8437-e08c5ac83db5-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:11.634602 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:11.634574    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"
Sep 11 13:50:11.634801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:11.634568    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j" event={"ID":"d12262da-8594-4f54-8437-e08c5ac83db5","Type":"ContainerDied","Data":"98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0"}
Sep 11 13:50:11.634801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:11.634703    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="98bf72614a00955b68c0d03c3a42a51653c6d22cb6c6d74af390d60cb168ebd0"
Sep 11 13:50:18.742154 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:18.742124    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:50:18.742569 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:18.742346    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" podUID="279acda3-5f66-48af-94dd-d6f3664f407b" containerName="payload-processing" containerID="cri-o://09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420" gracePeriod=30
Sep 11 13:50:18.758537 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:18.758502    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:50:18.758827 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:18.758774    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" podUID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" containerName="payload-pre-processing" containerID="cri-o://36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55" gracePeriod=30
Sep 11 13:50:19.020973 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.020893    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"]
Sep 11 13:50:19.027050 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.027013    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-cleanup-2e3865-kk87j"]
Sep 11 13:50:19.046269 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.046242    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:50:19.049171 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.049149    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:50:19.133760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.133728    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume\") pod \"279acda3-5f66-48af-94dd-d6f3664f407b\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") "
Sep 11 13:50:19.133760 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.133767    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume\") pod \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") "
Sep 11 13:50:19.134018 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.133792    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-qrvh6\" (UniqueName: \"kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6\") pod \"279acda3-5f66-48af-94dd-d6f3664f407b\" (UID: \"279acda3-5f66-48af-94dd-d6f3664f407b\") "
Sep 11 13:50:19.134018 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.133924    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-fw2kc\" (UniqueName: \"kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc\") pod \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\" (UID: \"c2534b7d-18bb-4b28-a989-ffb24f1b5936\") "
Sep 11 13:50:19.134197 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.134166    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "279acda3-5f66-48af-94dd-d6f3664f407b" (UID: "279acda3-5f66-48af-94dd-d6f3664f407b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:19.134274 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.134172    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "c2534b7d-18bb-4b28-a989-ffb24f1b5936" (UID: "c2534b7d-18bb-4b28-a989-ffb24f1b5936"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:19.134337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.134267    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/279acda3-5f66-48af-94dd-d6f3664f407b-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:19.134337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.134289    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c2534b7d-18bb-4b28-a989-ffb24f1b5936-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:19.136046 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.136021    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6" (OuterVolumeSpecName: "kube-api-access-qrvh6") pod "279acda3-5f66-48af-94dd-d6f3664f407b" (UID: "279acda3-5f66-48af-94dd-d6f3664f407b"). InnerVolumeSpecName "kube-api-access-qrvh6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:19.136188 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.136165    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc" (OuterVolumeSpecName: "kube-api-access-fw2kc") pod "c2534b7d-18bb-4b28-a989-ffb24f1b5936" (UID: "c2534b7d-18bb-4b28-a989-ffb24f1b5936"). InnerVolumeSpecName "kube-api-access-fw2kc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:19.235304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.235269    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrvh6\" (UniqueName: \"kubernetes.io/projected/279acda3-5f66-48af-94dd-d6f3664f407b-kube-api-access-qrvh6\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:19.235304 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.235298    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fw2kc\" (UniqueName: \"kubernetes.io/projected/c2534b7d-18bb-4b28-a989-ffb24f1b5936-kube-api-access-fw2kc\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:19.677169 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.677138    2551 generic.go:356] "Generic (PLEG): container finished" podID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" containerID="36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55" exitCode=0
Sep 11 13:50:19.677376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.677181    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"
Sep 11 13:50:19.677376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.677215    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" event={"ID":"c2534b7d-18bb-4b28-a989-ffb24f1b5936","Type":"ContainerDied","Data":"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"}
Sep 11 13:50:19.677376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.677268    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj" event={"ID":"c2534b7d-18bb-4b28-a989-ffb24f1b5936","Type":"ContainerDied","Data":"f15e8b93ceb22b2bdc5f3e3f7384a58c9a841616019a9ad62116fcd029b2e896"}
Sep 11 13:50:19.677376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.677299    2551 scope.go:117] "RemoveContainer" containerID="36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"
Sep 11 13:50:19.678779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.678757    2551 generic.go:356] "Generic (PLEG): container finished" podID="279acda3-5f66-48af-94dd-d6f3664f407b" containerID="09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420" exitCode=0
Sep 11 13:50:19.678882 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.678804    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"
Sep 11 13:50:19.678946 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.678803    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" event={"ID":"279acda3-5f66-48af-94dd-d6f3664f407b","Type":"ContainerDied","Data":"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"}
Sep 11 13:50:19.678946 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.678926    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz" event={"ID":"279acda3-5f66-48af-94dd-d6f3664f407b","Type":"ContainerDied","Data":"74ae123e42185ed8287c57dd51148c3d29758122c67b76f37ee923ee54a88ed2"}
Sep 11 13:50:19.687112 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.687094    2551 scope.go:117] "RemoveContainer" containerID="36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"
Sep 11 13:50:19.687451 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:50:19.687429    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55\": container with ID starting with 36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55 not found: ID does not exist" containerID="36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"
Sep 11 13:50:19.687531 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.687464    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55"} err="failed to get container status \"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55\": rpc error: code = NotFound desc = could not find container \"36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55\": container with ID starting with 36f7eac2a84a68e1c8a247c92117fffe9a41b80e41b0cef21e8544c10d7bdf55 not found: ID does not exist"
Sep 11 13:50:19.687531 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.687487    2551 scope.go:117] "RemoveContainer" containerID="09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"
Sep 11 13:50:19.695753 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.695734    2551 scope.go:117] "RemoveContainer" containerID="09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"
Sep 11 13:50:19.696032 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:50:19.696014    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420\": container with ID starting with 09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420 not found: ID does not exist" containerID="09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"
Sep 11 13:50:19.696076 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.696039    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420"} err="failed to get container status \"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420\": rpc error: code = NotFound desc = could not find container \"09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420\": container with ID starting with 09c9285ae4440dda6db289217507240fc895314c475d8be71023bb67db773420 not found: ID does not exist"
Sep 11 13:50:19.705675 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.705652    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:50:19.710259 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.710237    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-cleanup-2e3865-594d776b54-9q6kj"]
Sep 11 13:50:19.725674 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.725622    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:50:19.735282 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:19.735250    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-cleanup-2e3865-754889dd4f-68ckz"]
Sep 11 13:50:20.965033 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:20.964992    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="279acda3-5f66-48af-94dd-d6f3664f407b" path="/var/lib/kubelet/pods/279acda3-5f66-48af-94dd-d6f3664f407b/volumes"
Sep 11 13:50:20.965519 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:20.965499    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" path="/var/lib/kubelet/pods/c2534b7d-18bb-4b28-a989-ffb24f1b5936/volumes"
Sep 11 13:50:20.965914 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:20.965898    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d12262da-8594-4f54-8437-e08c5ac83db5" path="/var/lib/kubelet/pods/d12262da-8594-4f54-8437-e08c5ac83db5/volumes"
Sep 11 13:50:31.070250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070217    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"]
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070585    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="279acda3-5f66-48af-94dd-d6f3664f407b" containerName="payload-processing"
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070596    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="279acda3-5f66-48af-94dd-d6f3664f407b" containerName="payload-processing"
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070607    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d12262da-8594-4f54-8437-e08c5ac83db5" containerName="revoke-keys"
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070613    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d12262da-8594-4f54-8437-e08c5ac83db5" containerName="revoke-keys"
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070638    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" containerName="payload-pre-processing"
Sep 11 13:50:31.070679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070644    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" containerName="payload-pre-processing"
Sep 11 13:50:31.070891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070702    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c2534b7d-18bb-4b28-a989-ffb24f1b5936" containerName="payload-pre-processing"
Sep 11 13:50:31.070891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070710    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="279acda3-5f66-48af-94dd-d6f3664f407b" containerName="payload-processing"
Sep 11 13:50:31.070891 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.070716    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d12262da-8594-4f54-8437-e08c5ac83db5" containerName="revoke-keys"
Sep 11 13:50:31.074916 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.074898    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:31.084326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.084302    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:50:31.084433 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.084331    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:50:31.097531 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.097508    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"]
Sep 11 13:50:31.139189 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.139156    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2\" (UID: \"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:31.240359 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.240326    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2\" (UID: \"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:31.241010 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.240976    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2\" (UID: \"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:31.385074 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.384990    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:31.660235 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.660197    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"]
Sep 11 13:50:31.662477 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:50:31.662449    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf37ec34_4a80_4aa1_9e00_a36ae56a85b4.slice/crio-43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087 WatchSource:0}: Error finding container 43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087: Status 404 returned error can't find the container with id 43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087
Sep 11 13:50:31.743972 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:31.743938    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2" event={"ID":"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4","Type":"ContainerStarted","Data":"43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087"}
Sep 11 13:50:32.751079 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:32.751044    2551 generic.go:356] "Generic (PLEG): container finished" podID="cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" containerID="2389f295883752e11244a05f32c9d1b0a2d3a35b39c558d5505bf8b58dc28437" exitCode=0
Sep 11 13:50:32.751455 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:32.751098    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2" event={"ID":"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4","Type":"ContainerDied","Data":"2389f295883752e11244a05f32c9d1b0a2d3a35b39c558d5505bf8b58dc28437"}
Sep 11 13:50:33.887642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:33.887612    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:33.965585 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:33.965556    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca\") pod \"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4\" (UID: \"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4\") "
Sep 11 13:50:33.966017 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:33.965992    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" (UID: "cf37ec34-4a80-4aa1-9e00-a36ae56a85b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:34.067183 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:34.067097    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:34.763163 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:34.763135    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"
Sep 11 13:50:34.763376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:34.763162    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2" event={"ID":"cf37ec34-4a80-4aa1-9e00-a36ae56a85b4","Type":"ContainerDied","Data":"43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087"}
Sep 11 13:50:34.763376 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:34.763213    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="43ac07dde1422fe1bbf5264f5867950f8c6a1d4ffa1ad4183d19b96296454087"
Sep 11 13:50:47.299442 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.299409    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:50:47.299942 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.299700    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" podUID="06d50f02-87dc-4a80-80b7-9e35a20ff245" containerName="payload-processing" containerID="cri-o://83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f" gracePeriod=30
Sep 11 13:50:47.313338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.312830    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:50:47.313338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.313057    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" podUID="16743710-046e-4704-99b9-8220f32ca42c" containerName="payload-pre-processing" containerID="cri-o://78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39" gracePeriod=30
Sep 11 13:50:47.556672 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.556564    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"]
Sep 11 13:50:47.560090 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.560059    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-a-4920fa-76sn2"]
Sep 11 13:50:47.593802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.593780    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:50:47.596409 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.596389    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:50:47.690953 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.690922    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-kx2qw\" (UniqueName: \"kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw\") pod \"06d50f02-87dc-4a80-80b7-9e35a20ff245\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") "
Sep 11 13:50:47.691111 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.690983    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-tn992\" (UniqueName: \"kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992\") pod \"16743710-046e-4704-99b9-8220f32ca42c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") "
Sep 11 13:50:47.691111 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.691012    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume\") pod \"06d50f02-87dc-4a80-80b7-9e35a20ff245\" (UID: \"06d50f02-87dc-4a80-80b7-9e35a20ff245\") "
Sep 11 13:50:47.691111 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.691106    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume\") pod \"16743710-046e-4704-99b9-8220f32ca42c\" (UID: \"16743710-046e-4704-99b9-8220f32ca42c\") "
Sep 11 13:50:47.691483 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.691455    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "06d50f02-87dc-4a80-80b7-9e35a20ff245" (UID: "06d50f02-87dc-4a80-80b7-9e35a20ff245"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:47.691578 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.691495    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "16743710-046e-4704-99b9-8220f32ca42c" (UID: "16743710-046e-4704-99b9-8220f32ca42c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:50:47.693157 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.693128    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw" (OuterVolumeSpecName: "kube-api-access-kx2qw") pod "06d50f02-87dc-4a80-80b7-9e35a20ff245" (UID: "06d50f02-87dc-4a80-80b7-9e35a20ff245"). InnerVolumeSpecName "kube-api-access-kx2qw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:47.693250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.693134    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992" (OuterVolumeSpecName: "kube-api-access-tn992") pod "16743710-046e-4704-99b9-8220f32ca42c" (UID: "16743710-046e-4704-99b9-8220f32ca42c"). InnerVolumeSpecName "kube-api-access-tn992". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:50:47.792536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.792498    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/16743710-046e-4704-99b9-8220f32ca42c-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:47.792536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.792528    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kx2qw\" (UniqueName: \"kubernetes.io/projected/06d50f02-87dc-4a80-80b7-9e35a20ff245-kube-api-access-kx2qw\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:47.792536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.792536    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tn992\" (UniqueName: \"kubernetes.io/projected/16743710-046e-4704-99b9-8220f32ca42c-kube-api-access-tn992\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:47.792536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.792544    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/06d50f02-87dc-4a80-80b7-9e35a20ff245-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:50:47.832463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.832383    2551 generic.go:356] "Generic (PLEG): container finished" podID="06d50f02-87dc-4a80-80b7-9e35a20ff245" containerID="83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f" exitCode=0
Sep 11 13:50:47.832463 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.832430    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"
Sep 11 13:50:47.832671 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.832469    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" event={"ID":"06d50f02-87dc-4a80-80b7-9e35a20ff245","Type":"ContainerDied","Data":"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"}
Sep 11 13:50:47.832671 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.832524    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8" event={"ID":"06d50f02-87dc-4a80-80b7-9e35a20ff245","Type":"ContainerDied","Data":"662267bf84f7a96ea121d025c98b7c74d860adc55a64a8b1aa868637b41493b0"}
Sep 11 13:50:47.832671 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.832546    2551 scope.go:117] "RemoveContainer" containerID="83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"
Sep 11 13:50:47.837131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.837056    2551 generic.go:356] "Generic (PLEG): container finished" podID="16743710-046e-4704-99b9-8220f32ca42c" containerID="78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39" exitCode=0
Sep 11 13:50:47.837131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.837096    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" event={"ID":"16743710-046e-4704-99b9-8220f32ca42c","Type":"ContainerDied","Data":"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"}
Sep 11 13:50:47.837131 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.837121    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"
Sep 11 13:50:47.837361 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.837164    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c" event={"ID":"16743710-046e-4704-99b9-8220f32ca42c","Type":"ContainerDied","Data":"bebc3243a682cbd82c624b2384999b3ee12b1a49735bcf501cf103a591359844"}
Sep 11 13:50:47.843494 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.843470    2551 scope.go:117] "RemoveContainer" containerID="83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"
Sep 11 13:50:47.843824 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:50:47.843805    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f\": container with ID starting with 83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f not found: ID does not exist" containerID="83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"
Sep 11 13:50:47.843892 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.843833    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f"} err="failed to get container status \"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f\": rpc error: code = NotFound desc = could not find container \"83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f\": container with ID starting with 83c577c77215b0bb03d7ee11e0e51211435e3c5494033edbd545a001bf00509f not found: ID does not exist"
Sep 11 13:50:47.843892 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.843849    2551 scope.go:117] "RemoveContainer" containerID="78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"
Sep 11 13:50:47.854338 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.854320    2551 scope.go:117] "RemoveContainer" containerID="78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"
Sep 11 13:50:47.854859 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:50:47.854778    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39\": container with ID starting with 78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39 not found: ID does not exist" containerID="78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"
Sep 11 13:50:47.854859 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.854817    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39"} err="failed to get container status \"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39\": rpc error: code = NotFound desc = could not find container \"78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39\": container with ID starting with 78f959a5a5cf4938247623ae3e585a4b03370c377f317c966944013227f14f39 not found: ID does not exist"
Sep 11 13:50:47.856933 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.856910    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:50:47.860819 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.860797    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-a-4920fa-78d65d6589-4lvh8"]
Sep 11 13:50:47.872838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.872773    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:50:47.874373 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:47.874354    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-a-4920fa-5c6b68c698-jgt7c"]
Sep 11 13:50:48.964233 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:48.964199    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="06d50f02-87dc-4a80-80b7-9e35a20ff245" path="/var/lib/kubelet/pods/06d50f02-87dc-4a80-80b7-9e35a20ff245/volumes"
Sep 11 13:50:48.964608 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:48.964592    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="16743710-046e-4704-99b9-8220f32ca42c" path="/var/lib/kubelet/pods/16743710-046e-4704-99b9-8220f32ca42c/volumes"
Sep 11 13:50:48.964939 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:50:48.964924    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" path="/var/lib/kubelet/pods/cf37ec34-4a80-4aa1-9e00-a36ae56a85b4/volumes"
Sep 11 13:51:29.475927 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:29.475899    2551 scope.go:117] "RemoveContainer" containerID="d81920cf40f66e6f04a55a62bee2e4161458fae0be152f1b839c843abdab4ba4"
Sep 11 13:51:29.484207 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:29.484180    2551 scope.go:117] "RemoveContainer" containerID="573bfc5f8391fb68baa6bb9735a0a18266471ad12f0c4ba646d3de550085be58"
Sep 11 13:51:36.895751 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.895710    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"]
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896157    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" containerName="revoke-keys"
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896171    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" containerName="revoke-keys"
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896183    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="16743710-046e-4704-99b9-8220f32ca42c" containerName="payload-pre-processing"
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896189    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="16743710-046e-4704-99b9-8220f32ca42c" containerName="payload-pre-processing"
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896200    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="06d50f02-87dc-4a80-80b7-9e35a20ff245" containerName="payload-processing"
Sep 11 13:51:36.896231 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896205    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="06d50f02-87dc-4a80-80b7-9e35a20ff245" containerName="payload-processing"
Sep 11 13:51:36.896511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896262    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="16743710-046e-4704-99b9-8220f32ca42c" containerName="payload-pre-processing"
Sep 11 13:51:36.896511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896270    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="cf37ec34-4a80-4aa1-9e00-a36ae56a85b4" containerName="revoke-keys"
Sep 11 13:51:36.896511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.896276    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="06d50f02-87dc-4a80-80b7-9e35a20ff245" containerName="payload-processing"
Sep 11 13:51:36.899305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.899286    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:36.901457 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.901432    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:51:36.901571 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.901484    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:51:36.907982 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:36.907957    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"]
Sep 11 13:51:37.019753 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:37.019709    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht\" (UID: \"391c7b83-87ba-454c-a40b-3dc2e44205a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:37.121149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:37.121117    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht\" (UID: \"391c7b83-87ba-454c-a40b-3dc2e44205a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:37.121754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:37.121735    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht\" (UID: \"391c7b83-87ba-454c-a40b-3dc2e44205a2\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:37.209605 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:37.209519    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:37.485016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:37.484984    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"]
Sep 11 13:51:37.488386 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:51:37.488349    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod391c7b83_87ba_454c_a40b_3dc2e44205a2.slice/crio-d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c WatchSource:0}: Error finding container d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c: Status 404 returned error can't find the container with id d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c
Sep 11 13:51:38.091454 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:38.091375    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht" event={"ID":"391c7b83-87ba-454c-a40b-3dc2e44205a2","Type":"ContainerStarted","Data":"886e4bc59b3599abc7f4225e976080732bbeb64ab8844becef7d6ec82ac3dff5"}
Sep 11 13:51:38.091454 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:38.091428    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht" event={"ID":"391c7b83-87ba-454c-a40b-3dc2e44205a2","Type":"ContainerStarted","Data":"d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c"}
Sep 11 13:51:38.111051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:38.111005    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht" podStartSLOduration=2.110989931 podStartE2EDuration="2.110989931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:51:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:51:38.109076021 +0000 UTC m=+1929.609980461" watchObservedRunningTime="2026-09-11 13:51:38.110989931 +0000 UTC m=+1929.611894378"
Sep 11 13:51:39.097554 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:39.097515    2551 generic.go:356] "Generic (PLEG): container finished" podID="391c7b83-87ba-454c-a40b-3dc2e44205a2" containerID="886e4bc59b3599abc7f4225e976080732bbeb64ab8844becef7d6ec82ac3dff5" exitCode=0
Sep 11 13:51:39.097947 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:39.097597    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht" event={"ID":"391c7b83-87ba-454c-a40b-3dc2e44205a2","Type":"ContainerDied","Data":"886e4bc59b3599abc7f4225e976080732bbeb64ab8844becef7d6ec82ac3dff5"}
Sep 11 13:51:40.234591 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:40.234565    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:40.349746 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:40.349708    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca\") pod \"391c7b83-87ba-454c-a40b-3dc2e44205a2\" (UID: \"391c7b83-87ba-454c-a40b-3dc2e44205a2\") "
Sep 11 13:51:40.350081 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:40.350059    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "391c7b83-87ba-454c-a40b-3dc2e44205a2" (UID: "391c7b83-87ba-454c-a40b-3dc2e44205a2"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:51:40.451101 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:40.451020    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/391c7b83-87ba-454c-a40b-3dc2e44205a2-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:41.108016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:41.107984    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht" event={"ID":"391c7b83-87ba-454c-a40b-3dc2e44205a2","Type":"ContainerDied","Data":"d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c"}
Sep 11 13:51:41.108016 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:41.108019    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"
Sep 11 13:51:41.108226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:41.108032    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d1ebfc061ba1298d5307f0b258f66582ee46c9c3377fe9711b67afcc81cbd82c"
Sep 11 13:51:47.910585 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:47.910552    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:51:47.911008 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:47.910824    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerName="payload-processing" containerID="cri-o://57aa33943f094ac497efdf751308633525cdae90c07693a66d4aea5b2129ea3e" gracePeriod=30
Sep 11 13:51:47.931036 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:47.931006    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:51:47.931211 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:47.931191    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerName="payload-pre-processing" containerID="cri-o://0c814495c3caede130f817aa6411f65bb2fb67f9fecdf109155c415e91e24675" gracePeriod=30
Sep 11 13:51:48.147740 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.147531    2551 generic.go:356] "Generic (PLEG): container finished" podID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerID="57aa33943f094ac497efdf751308633525cdae90c07693a66d4aea5b2129ea3e" exitCode=0
Sep 11 13:51:48.147740 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.147698    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" event={"ID":"ac2a335b-f96d-482e-ab74-d6f371a50021","Type":"ContainerDied","Data":"57aa33943f094ac497efdf751308633525cdae90c07693a66d4aea5b2129ea3e"}
Sep 11 13:51:48.150134 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.150107    2551 generic.go:356] "Generic (PLEG): container finished" podID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerID="0c814495c3caede130f817aa6411f65bb2fb67f9fecdf109155c415e91e24675" exitCode=0
Sep 11 13:51:48.150261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.150184    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" event={"ID":"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b","Type":"ContainerDied","Data":"0c814495c3caede130f817aa6411f65bb2fb67f9fecdf109155c415e91e24675"}
Sep 11 13:51:48.182608 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.182586    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:51:48.210321 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.210293    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume\") pod \"ac2a335b-f96d-482e-ab74-d6f371a50021\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") "
Sep 11 13:51:48.210457 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.210396    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4gch\" (UniqueName: \"kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch\") pod \"ac2a335b-f96d-482e-ab74-d6f371a50021\" (UID: \"ac2a335b-f96d-482e-ab74-d6f371a50021\") "
Sep 11 13:51:48.210617 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.210593    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "ac2a335b-f96d-482e-ab74-d6f371a50021" (UID: "ac2a335b-f96d-482e-ab74-d6f371a50021"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:51:48.212481 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.212451    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch" (OuterVolumeSpecName: "kube-api-access-w4gch") pod "ac2a335b-f96d-482e-ab74-d6f371a50021" (UID: "ac2a335b-f96d-482e-ab74-d6f371a50021"). InnerVolumeSpecName "kube-api-access-w4gch". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:51:48.215601 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.215585    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:51:48.300203 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.300167    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"]
Sep 11 13:51:48.302593 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.302567    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ipp-b-9efb13-bnqht"]
Sep 11 13:51:48.311705 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.311680    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume\") pod \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") "
Sep 11 13:51:48.311846 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.311751    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-n7xln\" (UniqueName: \"kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln\") pod \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\" (UID: \"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b\") "
Sep 11 13:51:48.311912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.311886    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4gch\" (UniqueName: \"kubernetes.io/projected/ac2a335b-f96d-482e-ab74-d6f371a50021-kube-api-access-w4gch\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:48.311912 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.311897    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ac2a335b-f96d-482e-ab74-d6f371a50021-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:48.312104 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.312079    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" (UID: "4b7dc5fe-04c6-4f88-ab11-10a46a467c4b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:51:48.313769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.313748    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln" (OuterVolumeSpecName: "kube-api-access-n7xln") pod "4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" (UID: "4b7dc5fe-04c6-4f88-ab11-10a46a467c4b"). InnerVolumeSpecName "kube-api-access-n7xln". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:51:48.412861 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.412782    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n7xln\" (UniqueName: \"kubernetes.io/projected/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-kube-api-access-n7xln\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:48.412861 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.412809    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:48.964889 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:48.964853    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="391c7b83-87ba-454c-a40b-3dc2e44205a2" path="/var/lib/kubelet/pods/391c7b83-87ba-454c-a40b-3dc2e44205a2/volumes"
Sep 11 13:51:49.135254 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.135207    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.132.0.83:9004: i/o timeout"
Sep 11 13:51:49.135430 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.135207    2551 prober.go:120] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerName="payload-processing" probeResult="failure" output="dial tcp 10.132.0.84:9004: i/o timeout"
Sep 11 13:51:49.157228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.157195    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"
Sep 11 13:51:49.157228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.157218    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf" event={"ID":"ac2a335b-f96d-482e-ab74-d6f371a50021","Type":"ContainerDied","Data":"6e8485c7c1095dd3a2fa2192a94ec41b0bd8124fe5342c543c4e2e7c9927bd87"}
Sep 11 13:51:49.157469 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.157287    2551 scope.go:117] "RemoveContainer" containerID="57aa33943f094ac497efdf751308633525cdae90c07693a66d4aea5b2129ea3e"
Sep 11 13:51:49.158680 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.158657    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"
Sep 11 13:51:49.158780 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.158657    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8" event={"ID":"4b7dc5fe-04c6-4f88-ab11-10a46a467c4b","Type":"ContainerDied","Data":"a382f6ed47de5c1e6570b5b31ad9cd1110fba509b88d9240340cb813c3416d8c"}
Sep 11 13:51:49.165710 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.165690    2551 scope.go:117] "RemoveContainer" containerID="0c814495c3caede130f817aa6411f65bb2fb67f9fecdf109155c415e91e24675"
Sep 11 13:51:49.201199 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.201141    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:51:49.203250 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.203229    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ipp-b-9efb13-779b44b79f-8qlsf"]
Sep 11 13:51:49.226709 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.226686    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:51:49.240697 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:49.240671    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ipp-b-9efb13-767dc6d58b-cgls8"]
Sep 11 13:51:50.964558 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:50.964525    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" path="/var/lib/kubelet/pods/4b7dc5fe-04c6-4f88-ab11-10a46a467c4b/volumes"
Sep 11 13:51:50.964952 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:50.964894    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" path="/var/lib/kubelet/pods/ac2a335b-f96d-482e-ab74-d6f371a50021/volumes"
Sep 11 13:51:55.591240 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591198    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"]
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591542    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="391c7b83-87ba-454c-a40b-3dc2e44205a2" containerName="revoke-keys"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591552    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="391c7b83-87ba-454c-a40b-3dc2e44205a2" containerName="revoke-keys"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591564    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerName="payload-pre-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591569    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerName="payload-pre-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591574    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerName="payload-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591580    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerName="payload-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591654    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="ac2a335b-f96d-482e-ab74-d6f371a50021" containerName="payload-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591663    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="4b7dc5fe-04c6-4f88-ab11-10a46a467c4b" containerName="payload-pre-processing"
Sep 11 13:51:55.591841 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.591669    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="391c7b83-87ba-454c-a40b-3dc2e44205a2" containerName="revoke-keys"
Sep 11 13:51:55.595804 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.595788    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:55.598090 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.598052    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:51:55.598328 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.598313    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:51:55.606988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.606964    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"]
Sep 11 13:51:55.684866 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.684828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9\" (UID: \"09651fdd-2318-4c96-9d90-3d48ab02ebdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:55.785357 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.785321    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9\" (UID: \"09651fdd-2318-4c96-9d90-3d48ab02ebdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:55.785988 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.785969    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9\" (UID: \"09651fdd-2318-4c96-9d90-3d48ab02ebdc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:55.905802 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:55.905710    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:56.202100 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:51:56.202067    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod09651fdd_2318_4c96_9d90_3d48ab02ebdc.slice/crio-2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519 WatchSource:0}: Error finding container 2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519: Status 404 returned error can't find the container with id 2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519
Sep 11 13:51:56.202830 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:56.202805    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"]
Sep 11 13:51:57.205311 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:57.205279    2551 generic.go:356] "Generic (PLEG): container finished" podID="09651fdd-2318-4c96-9d90-3d48ab02ebdc" containerID="32beb69080a2276ab97981af0217e24fd967478c8f414b568147c7ea28e7048a" exitCode=0
Sep 11 13:51:57.205709 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:57.205362    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9" event={"ID":"09651fdd-2318-4c96-9d90-3d48ab02ebdc","Type":"ContainerDied","Data":"32beb69080a2276ab97981af0217e24fd967478c8f414b568147c7ea28e7048a"}
Sep 11 13:51:57.205709 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:57.205412    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9" event={"ID":"09651fdd-2318-4c96-9d90-3d48ab02ebdc","Type":"ContainerStarted","Data":"2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519"}
Sep 11 13:51:58.342200 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:58.342176    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:58.408164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:58.408135    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca\") pod \"09651fdd-2318-4c96-9d90-3d48ab02ebdc\" (UID: \"09651fdd-2318-4c96-9d90-3d48ab02ebdc\") "
Sep 11 13:51:58.408488 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:58.408465    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "09651fdd-2318-4c96-9d90-3d48ab02ebdc" (UID: "09651fdd-2318-4c96-9d90-3d48ab02ebdc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:51:58.509025 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:58.508989    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/09651fdd-2318-4c96-9d90-3d48ab02ebdc-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:51:59.223579 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:59.223506    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"
Sep 11 13:51:59.223750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:59.223506    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9" event={"ID":"09651fdd-2318-4c96-9d90-3d48ab02ebdc","Type":"ContainerDied","Data":"2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519"}
Sep 11 13:51:59.223750 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:51:59.223677    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2262945318e7754447a5faea802cd785eea3b00d4f895d0ce2c368492dfa5519"
Sep 11 13:52:06.172544 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.172510    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"]
Sep 11 13:52:06.177237 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.177210    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w3-3dd80c-29818905-fxnvv"]
Sep 11 13:52:06.201355 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.201323    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:52:06.201617 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.201588    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" podUID="66684dc5-1415-4550-bb36-af0b1a495906" containerName="payload-processing" containerID="cri-o://75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166" gracePeriod=30
Sep 11 13:52:06.209590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.209562    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:52:06.210401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.210352    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" podUID="76cad1b3-71f5-430e-9027-d4708c8416a9" containerName="payload-pre-processing" containerID="cri-o://107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee" gracePeriod=30
Sep 11 13:52:06.383580 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.383542    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"]
Sep 11 13:52:06.387477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.387418    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-3dd80c-c9mv9"]
Sep 11 13:52:06.446739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.446685    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:52:06.487830 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.487801    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:52:06.577711 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.577674    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-cq6vp\" (UniqueName: \"kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp\") pod \"76cad1b3-71f5-430e-9027-d4708c8416a9\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") "
Sep 11 13:52:06.577900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.577743    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-hlm7k\" (UniqueName: \"kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k\") pod \"66684dc5-1415-4550-bb36-af0b1a495906\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") "
Sep 11 13:52:06.577900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.577771    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume\") pod \"66684dc5-1415-4550-bb36-af0b1a495906\" (UID: \"66684dc5-1415-4550-bb36-af0b1a495906\") "
Sep 11 13:52:06.577900 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.577804    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume\") pod \"76cad1b3-71f5-430e-9027-d4708c8416a9\" (UID: \"76cad1b3-71f5-430e-9027-d4708c8416a9\") "
Sep 11 13:52:06.578164 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.578132    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "66684dc5-1415-4550-bb36-af0b1a495906" (UID: "66684dc5-1415-4550-bb36-af0b1a495906"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:06.578261 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.578211    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "76cad1b3-71f5-430e-9027-d4708c8416a9" (UID: "76cad1b3-71f5-430e-9027-d4708c8416a9"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:06.579807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.579788    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp" (OuterVolumeSpecName: "kube-api-access-cq6vp") pod "76cad1b3-71f5-430e-9027-d4708c8416a9" (UID: "76cad1b3-71f5-430e-9027-d4708c8416a9"). InnerVolumeSpecName "kube-api-access-cq6vp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:06.579921 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.579901    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k" (OuterVolumeSpecName: "kube-api-access-hlm7k") pod "66684dc5-1415-4550-bb36-af0b1a495906" (UID: "66684dc5-1415-4550-bb36-af0b1a495906"). InnerVolumeSpecName "kube-api-access-hlm7k". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:06.678507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.678419    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cq6vp\" (UniqueName: \"kubernetes.io/projected/76cad1b3-71f5-430e-9027-d4708c8416a9-kube-api-access-cq6vp\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:06.678507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.678451    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hlm7k\" (UniqueName: \"kubernetes.io/projected/66684dc5-1415-4550-bb36-af0b1a495906-kube-api-access-hlm7k\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:06.678507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.678459    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/66684dc5-1415-4550-bb36-af0b1a495906-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:06.678507 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.678468    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/76cad1b3-71f5-430e-9027-d4708c8416a9-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:06.964450 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.964371    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="09651fdd-2318-4c96-9d90-3d48ab02ebdc" path="/var/lib/kubelet/pods/09651fdd-2318-4c96-9d90-3d48ab02ebdc/volumes"
Sep 11 13:52:06.964671 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:06.964657    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="2d3c7897-17e0-4a65-bbac-96ff0dd0b299" path="/var/lib/kubelet/pods/2d3c7897-17e0-4a65-bbac-96ff0dd0b299/volumes"
Sep 11 13:52:07.268334 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.268295    2551 generic.go:356] "Generic (PLEG): container finished" podID="76cad1b3-71f5-430e-9027-d4708c8416a9" containerID="107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee" exitCode=0
Sep 11 13:52:07.268756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.268344    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"
Sep 11 13:52:07.268756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.268389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" event={"ID":"76cad1b3-71f5-430e-9027-d4708c8416a9","Type":"ContainerDied","Data":"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"}
Sep 11 13:52:07.268756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.268449    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm" event={"ID":"76cad1b3-71f5-430e-9027-d4708c8416a9","Type":"ContainerDied","Data":"d5440f73dc4fd589721c01557bb68c65d39bbe9cf1009b0366d9abe15a428072"}
Sep 11 13:52:07.268756 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.268480    2551 scope.go:117] "RemoveContainer" containerID="107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"
Sep 11 13:52:07.270015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.269995    2551 generic.go:356] "Generic (PLEG): container finished" podID="66684dc5-1415-4550-bb36-af0b1a495906" containerID="75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166" exitCode=0
Sep 11 13:52:07.270089 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.270027    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" event={"ID":"66684dc5-1415-4550-bb36-af0b1a495906","Type":"ContainerDied","Data":"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"}
Sep 11 13:52:07.270089 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.270046    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"
Sep 11 13:52:07.270089 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.270080    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4" event={"ID":"66684dc5-1415-4550-bb36-af0b1a495906","Type":"ContainerDied","Data":"7c76ce9a50053ad073026b9e1ff50c233e64372f3a4c74ad36f5548baa9ab7bf"}
Sep 11 13:52:07.277606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.277584    2551 scope.go:117] "RemoveContainer" containerID="107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"
Sep 11 13:52:07.277971 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:52:07.277950    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee\": container with ID starting with 107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee not found: ID does not exist" containerID="107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"
Sep 11 13:52:07.278059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.277982    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee"} err="failed to get container status \"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee\": rpc error: code = NotFound desc = could not find container \"107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee\": container with ID starting with 107667ad6aa00f90016d5088ed098f7ee45513a7576c55bdb587a7126566aaee not found: ID does not exist"
Sep 11 13:52:07.278059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.278000    2551 scope.go:117] "RemoveContainer" containerID="75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"
Sep 11 13:52:07.286160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.286126    2551 scope.go:117] "RemoveContainer" containerID="75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"
Sep 11 13:52:07.286452 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:52:07.286432    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166\": container with ID starting with 75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166 not found: ID does not exist" containerID="75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"
Sep 11 13:52:07.286515 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.286462    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166"} err="failed to get container status \"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166\": rpc error: code = NotFound desc = could not find container \"75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166\": container with ID starting with 75f78c9e5c13db83b327900307f75938d8082dcc8f99d419bce20e319ce73166 not found: ID does not exist"
Sep 11 13:52:07.292127 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.292099    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:52:07.299545 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.299520    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w3-3dd80c-5779bdb45f-tw2w4"]
Sep 11 13:52:07.316247 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.316216    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:52:07.319993 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:07.319968    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-3dd80c-f6cd96d6b-vnzzm"]
Sep 11 13:52:08.963794 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:08.963761    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="66684dc5-1415-4550-bb36-af0b1a495906" path="/var/lib/kubelet/pods/66684dc5-1415-4550-bb36-af0b1a495906/volumes"
Sep 11 13:52:08.964167 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:08.964110    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="76cad1b3-71f5-430e-9027-d4708c8416a9" path="/var/lib/kubelet/pods/76cad1b3-71f5-430e-9027-d4708c8416a9/volumes"
Sep 11 13:52:15.320412 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320374    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"]
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320705    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="09651fdd-2318-4c96-9d90-3d48ab02ebdc" containerName="revoke-keys"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320716    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="09651fdd-2318-4c96-9d90-3d48ab02ebdc" containerName="revoke-keys"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320728    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="66684dc5-1415-4550-bb36-af0b1a495906" containerName="payload-processing"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320734    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="66684dc5-1415-4550-bb36-af0b1a495906" containerName="payload-processing"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320745    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="76cad1b3-71f5-430e-9027-d4708c8416a9" containerName="payload-pre-processing"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320752    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="76cad1b3-71f5-430e-9027-d4708c8416a9" containerName="payload-pre-processing"
Sep 11 13:52:15.320799 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320801    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="76cad1b3-71f5-430e-9027-d4708c8416a9" containerName="payload-pre-processing"
Sep 11 13:52:15.321028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320810    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="66684dc5-1415-4550-bb36-af0b1a495906" containerName="payload-processing"
Sep 11 13:52:15.321028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.320819    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="09651fdd-2318-4c96-9d90-3d48ab02ebdc" containerName="revoke-keys"
Sep 11 13:52:15.325130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.325114    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:15.327666 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.327644    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-9tpvf\""
Sep 11 13:52:15.327783 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.327679    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Sep 11 13:52:15.337523 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.337496    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"]
Sep 11 13:52:15.456505 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.456467    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8\" (UID: \"c6717918-a411-4793-8860-bc768c5adfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:15.557095 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.557034    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8\" (UID: \"c6717918-a411-4793-8860-bc768c5adfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:15.557822 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.557797    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8\" (UID: \"c6717918-a411-4793-8860-bc768c5adfcc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:15.634983 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.634904    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:15.919887 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:15.919705    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"]
Sep 11 13:52:15.922558 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:52:15.922528    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc6717918_a411_4793_8860_bc768c5adfcc.slice/crio-f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c WatchSource:0}: Error finding container f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c: Status 404 returned error can't find the container with id f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c
Sep 11 13:52:16.316114 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:16.316077    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8" event={"ID":"c6717918-a411-4793-8860-bc768c5adfcc","Type":"ContainerStarted","Data":"399a7fdddd387c6b7c116c0fc8d3b611ab8fff7408d533bd5393b9c752a24621"}
Sep 11 13:52:16.316322 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:16.316136    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8" event={"ID":"c6717918-a411-4793-8860-bc768c5adfcc","Type":"ContainerStarted","Data":"f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c"}
Sep 11 13:52:16.339729 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:16.339681    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8" podStartSLOduration=1.339665634 podStartE2EDuration="1.339665634s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:52:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:52:16.338150814 +0000 UTC m=+1967.839055260" watchObservedRunningTime="2026-09-11 13:52:16.339665634 +0000 UTC m=+1967.840570088"
Sep 11 13:52:17.322521 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:17.322487    2551 generic.go:356] "Generic (PLEG): container finished" podID="c6717918-a411-4793-8860-bc768c5adfcc" containerID="399a7fdddd387c6b7c116c0fc8d3b611ab8fff7408d533bd5393b9c752a24621" exitCode=0
Sep 11 13:52:17.322687 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:17.322538    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8" event={"ID":"c6717918-a411-4793-8860-bc768c5adfcc","Type":"ContainerDied","Data":"399a7fdddd387c6b7c116c0fc8d3b611ab8fff7408d533bd5393b9c752a24621"}
Sep 11 13:52:18.455133 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:18.455105    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:18.583935 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:18.583835    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca\") pod \"c6717918-a411-4793-8860-bc768c5adfcc\" (UID: \"c6717918-a411-4793-8860-bc768c5adfcc\") "
Sep 11 13:52:18.584205 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:18.584182    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca" (OuterVolumeSpecName: "maas-api-service-ca") pod "c6717918-a411-4793-8860-bc768c5adfcc" (UID: "c6717918-a411-4793-8860-bc768c5adfcc"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:18.685375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:18.685341    2551 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/c6717918-a411-4793-8860-bc768c5adfcc-maas-api-service-ca\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:19.334452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:19.334364    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"
Sep 11 13:52:19.334452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:19.334389    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8" event={"ID":"c6717918-a411-4793-8860-bc768c5adfcc","Type":"ContainerDied","Data":"f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c"}
Sep 11 13:52:19.334452 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:19.334433    2551 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f548adf61bface65669e7ba9bb7afed82cf4242444f8b46245baee92edc7af5c"
Sep 11 13:52:26.152784 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.152707    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:52:26.153223 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.152932    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" podUID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" containerName="payload-processing" containerID="cri-o://329ea169fb8e7fe2f4bde8a7a4098e1a10ed98514ea5c39e5b8ed364f3ef8875" gracePeriod=30
Sep 11 13:52:26.159705 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.159673    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:52:26.159945 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.159905    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" podUID="d40b9629-03af-494a-83e8-e4bd04a2e66d" containerName="payload-pre-processing" containerID="cri-o://6bb503fef0f06d839b254a726ec6f8e6a3c90ebca1b1c012f27e35fb0f82fa01" gracePeriod=30
Sep 11 13:52:26.372955 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.372916    2551 generic.go:356] "Generic (PLEG): container finished" podID="d40b9629-03af-494a-83e8-e4bd04a2e66d" containerID="6bb503fef0f06d839b254a726ec6f8e6a3c90ebca1b1c012f27e35fb0f82fa01" exitCode=0
Sep 11 13:52:26.373097 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.372993    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" event={"ID":"d40b9629-03af-494a-83e8-e4bd04a2e66d","Type":"ContainerDied","Data":"6bb503fef0f06d839b254a726ec6f8e6a3c90ebca1b1c012f27e35fb0f82fa01"}
Sep 11 13:52:26.374765 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.374741    2551 generic.go:356] "Generic (PLEG): container finished" podID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" containerID="329ea169fb8e7fe2f4bde8a7a4098e1a10ed98514ea5c39e5b8ed364f3ef8875" exitCode=0
Sep 11 13:52:26.374914 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.374808    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" event={"ID":"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf","Type":"ContainerDied","Data":"329ea169fb8e7fe2f4bde8a7a4098e1a10ed98514ea5c39e5b8ed364f3ef8875"}
Sep 11 13:52:26.418977 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.418954    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:52:26.433713 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.433692    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:52:26.450366 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.450340    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"]
Sep 11 13:52:26.455243 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.455217    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-d6ca2b-dtxr8"]
Sep 11 13:52:26.556529 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.556494    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-mdn7v\" (UniqueName: \"kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v\") pod \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") "
Sep 11 13:52:26.556744 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.556540    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-w4wdv\" (UniqueName: \"kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv\") pod \"d40b9629-03af-494a-83e8-e4bd04a2e66d\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") "
Sep 11 13:52:26.556744 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.556577    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume\") pod \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\" (UID: \"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf\") "
Sep 11 13:52:26.556744 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.556687    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume\") pod \"d40b9629-03af-494a-83e8-e4bd04a2e66d\" (UID: \"d40b9629-03af-494a-83e8-e4bd04a2e66d\") "
Sep 11 13:52:26.557055 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.557025    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" (UID: "d5ad2cb2-94cb-43c0-8f2e-d822d70549cf"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:26.557182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.557153    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume" (OuterVolumeSpecName: "plugins-config-volume") pod "d40b9629-03af-494a-83e8-e4bd04a2e66d" (UID: "d40b9629-03af-494a-83e8-e4bd04a2e66d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Sep 11 13:52:26.558763 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.558731    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv" (OuterVolumeSpecName: "kube-api-access-w4wdv") pod "d40b9629-03af-494a-83e8-e4bd04a2e66d" (UID: "d40b9629-03af-494a-83e8-e4bd04a2e66d"). InnerVolumeSpecName "kube-api-access-w4wdv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:26.558881 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.558807    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v" (OuterVolumeSpecName: "kube-api-access-mdn7v") pod "d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" (UID: "d5ad2cb2-94cb-43c0-8f2e-d822d70549cf"). InnerVolumeSpecName "kube-api-access-mdn7v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:52:26.657397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.657305    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mdn7v\" (UniqueName: \"kubernetes.io/projected/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-kube-api-access-mdn7v\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:26.657397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.657341    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w4wdv\" (UniqueName: \"kubernetes.io/projected/d40b9629-03af-494a-83e8-e4bd04a2e66d-kube-api-access-w4wdv\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:26.657397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.657354    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:26.657397 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.657365    2551 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d40b9629-03af-494a-83e8-e4bd04a2e66d-plugins-config-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:52:26.966399 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:26.966323    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="c6717918-a411-4793-8860-bc768c5adfcc" path="/var/lib/kubelet/pods/c6717918-a411-4793-8860-bc768c5adfcc/volumes"
Sep 11 13:52:27.380779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.380750    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"
Sep 11 13:52:27.380779 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.380759    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5" event={"ID":"d5ad2cb2-94cb-43c0-8f2e-d822d70549cf","Type":"ContainerDied","Data":"c48b20d2b2129a6472d62538a89cd7d88c83d5f0c4cd44286e06a77f53782695"}
Sep 11 13:52:27.381326 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.380821    2551 scope.go:117] "RemoveContainer" containerID="329ea169fb8e7fe2f4bde8a7a4098e1a10ed98514ea5c39e5b8ed364f3ef8875"
Sep 11 13:52:27.382239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.382088    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"
Sep 11 13:52:27.382239 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.382110    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd" event={"ID":"d40b9629-03af-494a-83e8-e4bd04a2e66d","Type":"ContainerDied","Data":"00647a14d65d01b6a002afe04f590afe6eb2ed534a01f8653a11c52f297950ca"}
Sep 11 13:52:27.389807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.389790    2551 scope.go:117] "RemoveContainer" containerID="6bb503fef0f06d839b254a726ec6f8e6a3c90ebca1b1c012f27e35fb0f82fa01"
Sep 11 13:52:27.399660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.399616    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:52:27.405773 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.405753    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-b-w3-d6ca2b-784cff5d58-8wdxd"]
Sep 11 13:52:27.415290 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.415269    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:52:27.419509 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:27.419487    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-d6ca2b-694c7f464f-l5nk5"]
Sep 11 13:52:28.964488 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:28.964455    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d40b9629-03af-494a-83e8-e4bd04a2e66d" path="/var/lib/kubelet/pods/d40b9629-03af-494a-83e8-e4bd04a2e66d/volumes"
Sep 11 13:52:28.964853 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:28.964833    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" path="/var/lib/kubelet/pods/d5ad2cb2-94cb-43c0-8f2e-d822d70549cf/volumes"
Sep 11 13:52:29.513873 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:52:29.513841    2551 scope.go:117] "RemoveContainer" containerID="0801ab20c46caf568b60d0b2683586d06eaf40e86fd738a86571383093f35e38"
Sep 11 13:53:29.556533 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:29.556499    2551 scope.go:117] "RemoveContainer" containerID="cc8993a2c74b38296f8ee09a0d98affcf25a7330bc0d15d5e4496c8496cfe67d"
Sep 11 13:53:29.564440 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:29.564421    2551 scope.go:117] "RemoveContainer" containerID="97962b3ae8fb90bd6e8c67d939392ae72c114168cd77e3f9d84300d7ee37c032"
Sep 11 13:53:39.048867 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.048834    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049138    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" containerName="payload-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049149    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" containerName="payload-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049161    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="d40b9629-03af-494a-83e8-e4bd04a2e66d" containerName="payload-pre-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049166    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="d40b9629-03af-494a-83e8-e4bd04a2e66d" containerName="payload-pre-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049178    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="c6717918-a411-4793-8860-bc768c5adfcc" containerName="revoke-keys"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049183    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="c6717918-a411-4793-8860-bc768c5adfcc" containerName="revoke-keys"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049235    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d5ad2cb2-94cb-43c0-8f2e-d822d70549cf" containerName="payload-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049241    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="d40b9629-03af-494a-83e8-e4bd04a2e66d" containerName="payload-pre-processing"
Sep 11 13:53:39.049277 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.049249    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="c6717918-a411-4793-8860-bc768c5adfcc" containerName="revoke-keys"
Sep 11 13:53:39.051956 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.051936    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.055264 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.055244    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4q2dl\""
Sep 11 13:53:39.055801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.055780    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 13:53:39.055886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.055789    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 13:53:39.055886 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.055813    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 13:53:39.065792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.065767    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:53:39.152838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.152804    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.153027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.152845    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6qb69\" (UniqueName: \"kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.153027 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.152879    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.254162 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.254119    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6qb69\" (UniqueName: \"kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.254354 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.254174    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.254354 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.254263    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.256915 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.256882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.257040 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.256882    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.262583 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.262562    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-6qb69\" (UniqueName: \"kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69\") pod \"maas-controller-58558d56b9-zrkpw\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") " pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.361787 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.361692    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:39.633917 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.633712    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:53:39.636457 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:53:39.636429    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b85c4ae_6ec8_49c4_b4a7_58ed1eeace18.slice/crio-8c190afc02a8f78349f38ddaf7b3324fccc8b918fd356ab3e7262d4056a53f2e WatchSource:0}: Error finding container 8c190afc02a8f78349f38ddaf7b3324fccc8b918fd356ab3e7262d4056a53f2e: Status 404 returned error can't find the container with id 8c190afc02a8f78349f38ddaf7b3324fccc8b918fd356ab3e7262d4056a53f2e
Sep 11 13:53:39.637808 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.637789    2551 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Sep 11 13:53:39.726000 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:39.725969    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-zrkpw" event={"ID":"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18","Type":"ContainerStarted","Data":"8c190afc02a8f78349f38ddaf7b3324fccc8b918fd356ab3e7262d4056a53f2e"}
Sep 11 13:53:40.731769 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:40.731738    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-zrkpw" event={"ID":"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18","Type":"ContainerStarted","Data":"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"}
Sep 11 13:53:40.732146 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:40.731876    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:40.751015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:40.750970    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58558d56b9-zrkpw" podStartSLOduration=1.098839724 podStartE2EDuration="1.750956069s" totalImagesPullingTime="652.116345ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:53:39 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:53:40.749575802 +0000 UTC m=+2052.250480272" watchObservedRunningTime="2026-09-11 13:53:40.750956069 +0000 UTC m=+2052.251860518"
Sep 11 13:53:51.738937 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:51.738853    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:59.196938 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.196904    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:53:59.197347 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.197118    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="opendatahub/maas-controller-58558d56b9-zrkpw" podUID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" containerName="manager" containerID="cri-o://384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9" gracePeriod=30
Sep 11 13:53:59.446870 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.446841    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:59.529330 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.529299    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs\") pod \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") "
Sep 11 13:53:59.529536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.529356    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-6qb69\" (UniqueName: \"kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69\") pod \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") "
Sep 11 13:53:59.529536 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.529373    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert\") pod \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\" (UID: \"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18\") "
Sep 11 13:53:59.531462 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.531430    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert" (OuterVolumeSpecName: "webhook-cert") pod "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" (UID: "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18"). InnerVolumeSpecName "webhook-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:53:59.531885 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.531852    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs" (OuterVolumeSpecName: "metrics-certs") pod "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" (UID: "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:53:59.531983 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.531855    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69" (OuterVolumeSpecName: "kube-api-access-6qb69") pod "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" (UID: "7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18"). InnerVolumeSpecName "kube-api-access-6qb69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:53:59.630499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.630464    2551 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-metrics-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:53:59.630499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.630491    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6qb69\" (UniqueName: \"kubernetes.io/projected/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-kube-api-access-6qb69\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:53:59.630499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.630501    2551 reconciler_common.go:299] "Volume detached for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18-webhook-cert\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:53:59.823348 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.823258    2551 generic.go:356] "Generic (PLEG): container finished" podID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" containerID="384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9" exitCode=0
Sep 11 13:53:59.823348 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.823305    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-zrkpw"
Sep 11 13:53:59.823588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.823345    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-zrkpw" event={"ID":"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18","Type":"ContainerDied","Data":"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"}
Sep 11 13:53:59.823588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.823406    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-zrkpw" event={"ID":"7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18","Type":"ContainerDied","Data":"8c190afc02a8f78349f38ddaf7b3324fccc8b918fd356ab3e7262d4056a53f2e"}
Sep 11 13:53:59.823588 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.823437    2551 scope.go:117] "RemoveContainer" containerID="384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"
Sep 11 13:53:59.832573 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.832550    2551 scope.go:117] "RemoveContainer" containerID="384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"
Sep 11 13:53:59.832870 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:53:59.832850    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9\": container with ID starting with 384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9 not found: ID does not exist" containerID="384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"
Sep 11 13:53:59.832931 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.832878    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9"} err="failed to get container status \"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9\": rpc error: code = NotFound desc = could not find container \"384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9\": container with ID starting with 384638eec2c0fdabab82bf63131d47981b02b63dc01fb02636d12afe0a59e5c9 not found: ID does not exist"
Sep 11 13:53:59.850594 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.850555    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:53:59.856071 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:53:59.856042    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["opendatahub/maas-controller-58558d56b9-zrkpw"]
Sep 11 13:54:00.852690 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.852660    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["opendatahub/maas-controller-58558d56b9-n4skp"]
Sep 11 13:54:00.853280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.852980    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" containerName="manager"
Sep 11 13:54:00.853280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.852993    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" containerName="manager"
Sep 11 13:54:00.853280 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.853058    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" containerName="manager"
Sep 11 13:54:00.857125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.857103    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:00.859642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.859601    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"maas-parameters\""
Sep 11 13:54:00.860004 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.859985    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-dockercfg-4q2dl\""
Sep 11 13:54:00.860503 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.860480    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-metrics-tls\""
Sep 11 13:54:00.860774 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.860753    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"maas-controller-webhook-cert\""
Sep 11 13:54:00.867477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.867454    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58558d56b9-n4skp"]
Sep 11 13:54:00.942185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.942153    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-webhook-cert\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:00.942375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.942200    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-267wp\" (UniqueName: \"kubernetes.io/projected/d3019472-539f-428c-8a39-3f76e09cab76-kube-api-access-267wp\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:00.942375 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.942237    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-metrics-certs\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:00.964984 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:00.964954    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18" path="/var/lib/kubelet/pods/7b85c4ae-6ec8-49c4-b4a7-58ed1eeace18/volumes"
Sep 11 13:54:01.043529 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.043494    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-webhook-cert\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.043679 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.043540    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-267wp\" (UniqueName: \"kubernetes.io/projected/d3019472-539f-428c-8a39-3f76e09cab76-kube-api-access-267wp\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.043748 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.043729    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-metrics-certs\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.046087 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.046063    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-webhook-cert\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.046227 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.046066    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/d3019472-539f-428c-8a39-3f76e09cab76-metrics-certs\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.052530 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.052503    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-267wp\" (UniqueName: \"kubernetes.io/projected/d3019472-539f-428c-8a39-3f76e09cab76-kube-api-access-267wp\") pod \"maas-controller-58558d56b9-n4skp\" (UID: \"d3019472-539f-428c-8a39-3f76e09cab76\") " pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.168401 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.168311    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:01.459028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.459004    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["opendatahub/maas-controller-58558d56b9-n4skp"]
Sep 11 13:54:01.461227 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:54:01.461195    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd3019472_539f_428c_8a39_3f76e09cab76.slice/crio-f8c51bf3192f178b7b6bf51d8fbb465b29dcf5acf255a6e309b3582861b5c995 WatchSource:0}: Error finding container f8c51bf3192f178b7b6bf51d8fbb465b29dcf5acf255a6e309b3582861b5c995: Status 404 returned error can't find the container with id f8c51bf3192f178b7b6bf51d8fbb465b29dcf5acf255a6e309b3582861b5c995
Sep 11 13:54:01.841887 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:01.841860    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-n4skp" event={"ID":"d3019472-539f-428c-8a39-3f76e09cab76","Type":"ContainerStarted","Data":"f8c51bf3192f178b7b6bf51d8fbb465b29dcf5acf255a6e309b3582861b5c995"}
Sep 11 13:54:02.848690 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:02.848651    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="opendatahub/maas-controller-58558d56b9-n4skp" event={"ID":"d3019472-539f-428c-8a39-3f76e09cab76","Type":"ContainerStarted","Data":"3a5f0c43b421819f2cce9e5ea8007f4abc75b9a5e94b6e8b9981a638df4de8d6"}
Sep 11 13:54:02.849092 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:02.848720    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:02.870791 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:02.870737    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="opendatahub/maas-controller-58558d56b9-n4skp" podStartSLOduration=2.50268448 podStartE2EDuration="2.870719899s" totalImagesPullingTime="368.035419ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:54:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:54:02.869447062 +0000 UTC m=+2074.370351529" watchObservedRunningTime="2026-09-11 13:54:02.870719899 +0000 UTC m=+2074.371624348"
Sep 11 13:54:13.855793 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:13.855759    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/maas-controller-58558d56b9-n4skp"
Sep 11 13:54:29.586562 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:29.586530    2551 scope.go:117] "RemoveContainer" containerID="27869d90b3d37f4bf1ab8229e1c5d4bc85f58a2eeaf1c2eda0d54d3ee5f48f24"
Sep 11 13:54:29.594589 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:54:29.594570    2551 scope.go:117] "RemoveContainer" containerID="c6e887735e044d0b8ea75d68429dc8c0ff960346f5101a6e02d8aeb70690925e"
Sep 11 13:55:07.898798 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:07.898763    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:55:07.899324 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:07.898952    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" podUID="b50668f2-df96-4ef8-82f9-5729217169ae" containerName="manager" containerID="cri-o://b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5" gracePeriod=10
Sep 11 13:55:09.054883 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.054858    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:55:09.171421 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.171338    2551 generic.go:356] "Generic (PLEG): container finished" podID="b50668f2-df96-4ef8-82f9-5729217169ae" containerID="b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5" exitCode=0
Sep 11 13:55:09.171421 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.171382    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"
Sep 11 13:55:09.171590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.171419    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" event={"ID":"b50668f2-df96-4ef8-82f9-5729217169ae","Type":"ContainerDied","Data":"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"}
Sep 11 13:55:09.171590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.171478    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58" event={"ID":"b50668f2-df96-4ef8-82f9-5729217169ae","Type":"ContainerDied","Data":"672fdd666492e7e2fe3c5b7723c6c47f1c1383f297096b0077d8603c7475f299"}
Sep 11 13:55:09.171590 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.171499    2551 scope.go:117] "RemoveContainer" containerID="b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"
Sep 11 13:55:09.180182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.180167    2551 scope.go:117] "RemoveContainer" containerID="b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"
Sep 11 13:55:09.180433 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:55:09.180415    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5\": container with ID starting with b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5 not found: ID does not exist" containerID="b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"
Sep 11 13:55:09.180508 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.180441    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5"} err="failed to get container status \"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5\": rpc error: code = NotFound desc = could not find container \"b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5\": container with ID starting with b1f32f9623761139386b1f0357016485d4a33027365ba8203e61685ae751bfd5 not found: ID does not exist"
Sep 11 13:55:09.210778 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.210754    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-sc7zd\" (UniqueName: \"kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd\") pod \"b50668f2-df96-4ef8-82f9-5729217169ae\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") "
Sep 11 13:55:09.210876 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.210792    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume\") pod \"b50668f2-df96-4ef8-82f9-5729217169ae\" (UID: \"b50668f2-df96-4ef8-82f9-5729217169ae\") "
Sep 11 13:55:09.211149 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.211129    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume" (OuterVolumeSpecName: "extensions-socket-volume") pod "b50668f2-df96-4ef8-82f9-5729217169ae" (UID: "b50668f2-df96-4ef8-82f9-5729217169ae"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:55:09.212838 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.212820    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd" (OuterVolumeSpecName: "kube-api-access-sc7zd") pod "b50668f2-df96-4ef8-82f9-5729217169ae" (UID: "b50668f2-df96-4ef8-82f9-5729217169ae"). InnerVolumeSpecName "kube-api-access-sc7zd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:55:09.312193 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.312156    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sc7zd\" (UniqueName: \"kubernetes.io/projected/b50668f2-df96-4ef8-82f9-5729217169ae-kube-api-access-sc7zd\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:55:09.312193 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.312186    2551 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/b50668f2-df96-4ef8-82f9-5729217169ae-extensions-socket-volume\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:55:09.515736 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.515701    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:55:09.520160 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:09.520137    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-9lc58"]
Sep 11 13:55:10.963754 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:55:10.963722    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="b50668f2-df96-4ef8-82f9-5729217169ae" path="/var/lib/kubelet/pods/b50668f2-df96-4ef8-82f9-5729217169ae/volumes"
Sep 11 13:56:08.605213 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.605180    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"]
Sep 11 13:56:08.605660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.605504    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="b50668f2-df96-4ef8-82f9-5729217169ae" containerName="manager"
Sep 11 13:56:08.605660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.605518    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="b50668f2-df96-4ef8-82f9-5729217169ae" containerName="manager"
Sep 11 13:56:08.605660 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.605585    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="b50668f2-df96-4ef8-82f9-5729217169ae" containerName="manager"
Sep 11 13:56:08.608538 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.608523    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.623816 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.623795    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-m7xkw\""
Sep 11 13:56:08.660260 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.660235    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"]
Sep 11 13:56:08.679854 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.679828    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fwkqn\" (UniqueName: \"kubernetes.io/projected/95f74c87-76ab-42f3-ab0a-2b71142e2d24-kube-api-access-fwkqn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.679992 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.679871    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f74c87-76ab-42f3-ab0a-2b71142e2d24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.781069 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.781036    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fwkqn\" (UniqueName: \"kubernetes.io/projected/95f74c87-76ab-42f3-ab0a-2b71142e2d24-kube-api-access-fwkqn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.781265 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.781099    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f74c87-76ab-42f3-ab0a-2b71142e2d24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.781421 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.781402    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/95f74c87-76ab-42f3-ab0a-2b71142e2d24-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.790558 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.790533    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-fwkqn\" (UniqueName: \"kubernetes.io/projected/95f74c87-76ab-42f3-ab0a-2b71142e2d24-kube-api-access-fwkqn\") pod \"kuadrant-operator-controller-manager-5769f99d9f-gnjkf\" (UID: \"95f74c87-76ab-42f3-ab0a-2b71142e2d24\") " pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:08.917981 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:08.917884    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:09.193070 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:09.193048    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"]
Sep 11 13:56:09.195982 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:56:09.195952    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod95f74c87_76ab_42f3_ab0a_2b71142e2d24.slice/crio-1d95762f972f475ce5a08f7352c57f734fdb1cc6298716317968acc2f829d697 WatchSource:0}: Error finding container 1d95762f972f475ce5a08f7352c57f734fdb1cc6298716317968acc2f829d697: Status 404 returned error can't find the container with id 1d95762f972f475ce5a08f7352c57f734fdb1cc6298716317968acc2f829d697
Sep 11 13:56:09.445437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:09.445342    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf" event={"ID":"95f74c87-76ab-42f3-ab0a-2b71142e2d24","Type":"ContainerStarted","Data":"25dfb0f289d96622f1f2a204471f2adcc8f4bfc00f980d77db8ddc381cf2fefc"}
Sep 11 13:56:09.445437 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:09.445393    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf" event={"ID":"95f74c87-76ab-42f3-ab0a-2b71142e2d24","Type":"ContainerStarted","Data":"1d95762f972f475ce5a08f7352c57f734fdb1cc6298716317968acc2f829d697"}
Sep 11 13:56:09.445721 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:09.445436    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:09.473340 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:09.473283    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf" podStartSLOduration=1.473261624 podStartE2EDuration="1.473261624s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:09.471393184 +0000 UTC m=+2200.972297652" watchObservedRunningTime="2026-09-11 13:56:09.473261624 +0000 UTC m=+2200.974166073"
Sep 11 13:56:20.453792 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:20.453758    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-5769f99d9f-gnjkf"
Sep 11 13:56:29.628307 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:29.628273    2551 scope.go:117] "RemoveContainer" containerID="1ee885a0fadd0be8a3dc694a8464975c03e2d8189fece49766e4abc26eaece27"
Sep 11 13:56:46.894042 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:46.894010    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:56:46.897786 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:46.897760    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:46.900130 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:46.900106    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-a-9fe19763-kserve-self-signed-certs\""
Sep 11 13:56:46.909271 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:46.909248    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:56:47.013041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.012994    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c2dk9\" (UniqueName: \"kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.013226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.013046    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.013226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.013166    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.013226 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.013207    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.013410 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.013303    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.013410 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.013339    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114445 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114408    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114455    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114573    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114655 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114602    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114836 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114679    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c2dk9\" (UniqueName: \"kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114836 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114715    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.114939 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114879    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.115005 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114988    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.115051 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.114983    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.116822 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.116801    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.116997 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.116979    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.124030 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.124007    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-c2dk9\" (UniqueName: \"kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9\") pod \"e2e-same-id-a-9fe19763-kserve-596595d575-8twrg\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") " pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.209657 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.209543    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:47.483519 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.483494    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:56:47.486110 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:56:47.486070    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod8db62d26_dbd7_439d_aed3_60acb97a7a68.slice/crio-386df9b6ca90018613f6d3f7707841d6d839175e3ab95b2267e8fca0ec5effbd WatchSource:0}: Error finding container 386df9b6ca90018613f6d3f7707841d6d839175e3ab95b2267e8fca0ec5effbd: Status 404 returned error can't find the container with id 386df9b6ca90018613f6d3f7707841d6d839175e3ab95b2267e8fca0ec5effbd
Sep 11 13:56:47.622689 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.622652    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" event={"ID":"8db62d26-dbd7-439d-aed3-60acb97a7a68","Type":"ContainerStarted","Data":"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"}
Sep 11 13:56:47.622894 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.622701    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" event={"ID":"8db62d26-dbd7-439d-aed3-60acb97a7a68","Type":"ContainerStarted","Data":"386df9b6ca90018613f6d3f7707841d6d839175e3ab95b2267e8fca0ec5effbd"}
Sep 11 13:56:47.641028 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:47.640975    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" podStartSLOduration=1.640959185 podStartE2EDuration="1.640959185s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-09-11 13:56:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:56:47.639069068 +0000 UTC m=+2239.139973516" watchObservedRunningTime="2026-09-11 13:56:47.640959185 +0000 UTC m=+2239.141863633"
Sep 11 13:56:57.210464 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:57.210382    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:57.210925 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:57.210655    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:57.223088 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:57.223060    2551 kubelet.go:2654] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:56:57.690474 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:56:57.690445    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:57:29.475782 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:29.475751    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:57:29.646927 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:29.646891    2551 scope.go:117] "RemoveContainer" containerID="2389f295883752e11244a05f32c9d1b0a2d3a35b39c558d5505bf8b58dc28437"
Sep 11 13:57:44.494879 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.494836    2551 kuberuntime_container.go:873] "Killing container with a grace period" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" podUID="8db62d26-dbd7-439d-aed3-60acb97a7a68" containerName="main" containerID="cri-o://c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954" gracePeriod=45
Sep 11 13:57:44.745181 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.745115    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:57:44.820772 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820735    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.820967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820799    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.820967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820860    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kube-api-access-c2dk9\" (UniqueName: \"kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.820967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820893    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.820967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820908    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.820967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.820929    2551 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs\") pod \"8db62d26-dbd7-439d-aed3-60acb97a7a68\" (UID: \"8db62d26-dbd7-439d-aed3-60acb97a7a68\") "
Sep 11 13:57:44.821228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.821064    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache" (OuterVolumeSpecName: "model-cache") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:57:44.821228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.821110    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir" (OuterVolumeSpecName: "tmp-dir") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:57:44.821228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.821169    2551 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-model-cache\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.821228 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.821193    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home" (OuterVolumeSpecName: "home") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:57:44.823182 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.823150    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs" (OuterVolumeSpecName: "tls-certs") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Sep 11 13:57:44.823305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.823180    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm" (OuterVolumeSpecName: "dshm") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Sep 11 13:57:44.823305 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.823191    2551 operation_generator.go:782] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9" (OuterVolumeSpecName: "kube-api-access-c2dk9") pod "8db62d26-dbd7-439d-aed3-60acb97a7a68" (UID: "8db62d26-dbd7-439d-aed3-60acb97a7a68"). InnerVolumeSpecName "kube-api-access-c2dk9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Sep 11 13:57:44.906659 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.906609    2551 generic.go:356] "Generic (PLEG): container finished" podID="8db62d26-dbd7-439d-aed3-60acb97a7a68" containerID="c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954" exitCode=0
Sep 11 13:57:44.906659 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.906662    2551 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"
Sep 11 13:57:44.906899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.906660    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" event={"ID":"8db62d26-dbd7-439d-aed3-60acb97a7a68","Type":"ContainerDied","Data":"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"}
Sep 11 13:57:44.906899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.906710    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg" event={"ID":"8db62d26-dbd7-439d-aed3-60acb97a7a68","Type":"ContainerDied","Data":"386df9b6ca90018613f6d3f7707841d6d839175e3ab95b2267e8fca0ec5effbd"}
Sep 11 13:57:44.906899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.906729    2551 scope.go:117] "RemoveContainer" containerID="c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"
Sep 11 13:57:44.916185 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.916167    2551 scope.go:117] "RemoveContainer" containerID="c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"
Sep 11 13:57:44.916440 ip-10-0-141-50 kubenswrapper[2551]: E0911 13:57:44.916422    2551 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954\": container with ID starting with c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954 not found: ID does not exist" containerID="c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"
Sep 11 13:57:44.916496 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.916449    2551 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954"} err="failed to get container status \"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954\": rpc error: code = NotFound desc = could not find container \"c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954\": container with ID starting with c522bfee2a98e136b9f0b7bf2e143ae2ec02df53efaf07b5530895c7e20d6954 not found: ID does not exist"
Sep 11 13:57:44.921499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.921480    2551 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c2dk9\" (UniqueName: \"kubernetes.io/projected/8db62d26-dbd7-439d-aed3-60acb97a7a68-kube-api-access-c2dk9\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.921499 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.921499    2551 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-dshm\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.921656 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.921510    2551 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-home\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.921656 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.921517    2551 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/8db62d26-dbd7-439d-aed3-60acb97a7a68-tls-certs\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.921656 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.921526    2551 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/8db62d26-dbd7-439d-aed3-60acb97a7a68-tmp-dir\") on node \"ip-10-0-141-50.ec2.internal\" DevicePath \"\""
Sep 11 13:57:44.932657 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.932622    2551 kubelet.go:2551] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:57:44.943015 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.942989    2551 kubelet.go:2545] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-a-9fe19763-kserve-596595d575-8twrg"]
Sep 11 13:57:44.965476 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:57:44.965447    2551 kubelet_volumes.go:163] "Cleaned up orphaned pod volumes dir" podUID="8db62d26-dbd7-439d-aed3-60acb97a7a68" path="/var/lib/kubelet/pods/8db62d26-dbd7-439d-aed3-60acb97a7a68/volumes"
Sep 11 13:58:29.588801 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.588724    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gkkmt_cd6ef683-4bf0-4601-97df-f5760a05cf62/manager/0.log"
Sep 11 13:58:29.665608 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.665579    2551 scope.go:117] "RemoveContainer" containerID="399a7fdddd387c6b7c116c0fc8d3b611ab8fff7408d533bd5393b9c752a24621"
Sep 11 13:58:29.673511 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.673488    2551 scope.go:117] "RemoveContainer" containerID="886e4bc59b3599abc7f4225e976080732bbeb64ab8844becef7d6ec82ac3dff5"
Sep 11 13:58:29.680765 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.680748    2551 scope.go:117] "RemoveContainer" containerID="32beb69080a2276ab97981af0217e24fd967478c8f414b568147c7ea28e7048a"
Sep 11 13:58:29.719990 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.719967    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58558d56b9-n4skp_d3019472-539f-428c-8a39-3f76e09cab76/manager/0.log"
Sep 11 13:58:29.835993 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:29.835966    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zdqvf_b496961b-a099-47ec-8a34-de2690f7aefd/server/0.log"
Sep 11 13:58:30.251602 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:30.251576    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64959b9bbc-flljd_56a7e15d-c3f0-4460-be97-ee7dc88886a9/copy-manifests/0.log"
Sep 11 13:58:30.258550 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:30.258528    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64959b9bbc-flljd_56a7e15d-c3f0-4460-be97-ee7dc88886a9/manager/0.log"
Sep 11 13:58:32.436805 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:32.436774    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jw59x_dc5ed2da-05ed-4b48-b322-e84b69bfe6e1/manager/0.log"
Sep 11 13:58:32.699997 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:32.699921    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gnjkf_95f74c87-76ab-42f3-ab0a-2b71142e2d24/manager/0.log"
Sep 11 13:58:32.966059 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:32.965977    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rpfwg_56080ba9-e22a-48f7-89ab-6318e6f843e1/manager/0.log"
Sep 11 13:58:33.494993 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:33.494962    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dbrjc_66485700-96fb-43e7-b7ce-48c63f234d47/discovery/0.log"
Sep 11 13:58:33.750129 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:33.750042    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8589c48f5d-9ldlx_56cec59c-ac3b-4480-aed5-2ec531d5ed27/kube-auth-proxy/0.log"
Sep 11 13:58:33.997171 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:33.997146    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k7gg4_c2f407a1-0426-4f25-bea0-9aa10bbc6e14/payload-pre-processing/0.log"
Sep 11 13:58:34.120225 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:34.120150    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"
Sep 11 13:58:34.746606 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:34.746575    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-simulated-kserve-59975d69f9-bfpdr_eec359d7-9c83-4116-9654-0a7541d9f843/main/0.log"
Sep 11 13:58:35.143317 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:35.143238    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-unconfigured-facebook-opt-125m-simulated-kserve-85b68fn75qn_77d024c1-df3b-47c9-a905-c64ec06abfac/main/0.log"
Sep 11 13:58:45.686936 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:45.686906    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-zgb2x_06617998-5730-4981-9d6d-1ccbc62c7ab5/global-pull-secret-syncer/0.log"
Sep 11 13:58:45.783619 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:45.783590    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-6z4jd_b863f06e-24b9-4893-80f4-d6923e658891/konnectivity-agent/0.log"
Sep 11 13:58:45.915849 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:45.915821    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-50.ec2.internal_eb3cc3d67cb8f0daa2eb5dc040e5c1e9/haproxy/0.log"
Sep 11 13:58:50.922033 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:50.922002    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_dns-operator-controller-manager-7b84cd7c9c-jw59x_dc5ed2da-05ed-4b48-b322-e84b69bfe6e1/manager/0.log"
Sep 11 13:58:51.010173 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:51.010134    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-5769f99d9f-gnjkf_95f74c87-76ab-42f3-ab0a-2b71142e2d24/manager/0.log"
Sep 11 13:58:51.158300 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:51.158254    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_limitador-operator-controller-manager-74495c69f8-rpfwg_56080ba9-e22a-48f7-89ab-6318e6f843e1/manager/0.log"
Sep 11 13:58:53.134885 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:53.134857    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pbd8_61d61321-ce79-40ab-b0ef-232eeadb4c3c/node-exporter/0.log"
Sep 11 13:58:53.165125 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:53.165100    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pbd8_61d61321-ce79-40ab-b0ef-232eeadb4c3c/kube-rbac-proxy/0.log"
Sep 11 13:58:53.191096 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:53.191072    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-9pbd8_61d61321-ce79-40ab-b0ef-232eeadb4c3c/init-textfile/0.log"
Sep 11 13:58:55.988967 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.988928    2551 kubelet.go:2535] "SyncLoop ADD" source="api" pods=["openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"]
Sep 11 13:58:55.989428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.989383    2551 cpu_manager.go:401] "RemoveStaleState: containerMap: removing container" podUID="8db62d26-dbd7-439d-aed3-60acb97a7a68" containerName="main"
Sep 11 13:58:55.989428 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.989400    2551 state_mem.go:107] "Deleted CPUSet assignment" podUID="8db62d26-dbd7-439d-aed3-60acb97a7a68" containerName="main"
Sep 11 13:58:55.989541 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.989499    2551 memory_manager.go:368] "RemoveStaleState removing state" podUID="8db62d26-dbd7-439d-aed3-60acb97a7a68" containerName="main"
Sep 11 13:58:55.992504 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.992483    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:55.996274 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.996250    2551 reflector.go:439] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-9fc22\"/\"default-dockercfg-jx2lb\""
Sep 11 13:58:55.996274 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.996263    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9fc22\"/\"kube-root-ca.crt\""
Sep 11 13:58:55.996521 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:55.996499    2551 reflector.go:439] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-9fc22\"/\"openshift-service-ca.crt\""
Sep 11 13:58:56.014491 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.014465    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"]
Sep 11 13:58:56.119241 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.119187    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-sys\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.119241 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.119251    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-podres\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.119477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.119365    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-lib-modules\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.119477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.119402    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9x24l\" (UniqueName: \"kubernetes.io/projected/81b956a1-8599-4c8f-aedc-9b54f628aa66-kube-api-access-9x24l\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.119477 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.119427    2551 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-proc\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220434 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220383    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-lib-modules\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220434 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220439    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9x24l\" (UniqueName: \"kubernetes.io/projected/81b956a1-8599-4c8f-aedc-9b54f628aa66-kube-api-access-9x24l\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220472    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-proc\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220500    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-sys\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220529    2551 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-podres\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-lib-modules\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220591    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-proc\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220621    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-sys\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.220739 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.220670    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/81b956a1-8599-4c8f-aedc-9b54f628aa66-podres\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.229596 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.229569    2551 operation_generator.go:616] "MountVolume.SetUp succeeded for volume \"kube-api-access-9x24l\" (UniqueName: \"kubernetes.io/projected/81b956a1-8599-4c8f-aedc-9b54f628aa66-kube-api-access-9x24l\") pod \"perf-node-gather-daemonset-gdx2w\" (UID: \"81b956a1-8599-4c8f-aedc-9b54f628aa66\") " pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.301762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.301669    2551 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:56.582874 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:56.582800    2551 kubelet.go:2542] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"]
Sep 11 13:58:56.586201 ip-10-0-141-50 kubenswrapper[2551]: W0911 13:58:56.586175    2551 manager.go:1169] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-pod81b956a1_8599_4c8f_aedc_9b54f628aa66.slice/crio-54062e71d4f01fa632624e519339a43dc515f388c4d05f51360012d531b3a6d2 WatchSource:0}: Error finding container 54062e71d4f01fa632624e519339a43dc515f388c4d05f51360012d531b3a6d2: Status 404 returned error can't find the container with id 54062e71d4f01fa632624e519339a43dc515f388c4d05f51360012d531b3a6d2
Sep 11 13:58:57.236240 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:57.236201    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w" event={"ID":"81b956a1-8599-4c8f-aedc-9b54f628aa66","Type":"ContainerStarted","Data":"95fba9f8d5bd2e9d2605d38338d101ced3c2b7216cd94087ea00d0ba8f56081d"}
Sep 11 13:58:57.236713 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:57.236261    2551 kubelet.go:2566] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w" event={"ID":"81b956a1-8599-4c8f-aedc-9b54f628aa66","Type":"ContainerStarted","Data":"54062e71d4f01fa632624e519339a43dc515f388c4d05f51360012d531b3a6d2"}
Sep 11 13:58:57.236713 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:57.236300    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:58:57.254281 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:57.254228    2551 pod_startup_latency_tracker.go:144] "Observed pod startup duration" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w" podStartSLOduration=2.254212244 podStartE2EDuration="2.254212244s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-09-11 13:58:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-09-11 13:58:57.252298315 +0000 UTC m=+2368.753202784" watchObservedRunningTime="2026-09-11 13:58:57.254212244 +0000 UTC m=+2368.755116692"
Sep 11 13:58:58.628509 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:58.628483    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7lsz_d44b0dc9-4ee6-459e-98fb-a82a19841e55/dns/0.log"
Sep 11 13:58:58.659899 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:58.659871    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-p7lsz_d44b0dc9-4ee6-459e-98fb-a82a19841e55/kube-rbac-proxy/0.log"
Sep 11 13:58:58.823537 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:58.823509    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-gh9gl_817a83e4-76b4-4ac1-8afe-27c3beed2530/dns-node-resolver/0.log"
Sep 11 13:58:59.515054 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:58:59.515022    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-9wzjp_ee0bb0ee-3a05-4c9c-8191-0a9c1879af2c/node-ca/0.log"
Sep 11 13:59:00.955762 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:00.955732    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dbrjc_66485700-96fb-43e7-b7ce-48c63f234d47/discovery/0.log"
Sep 11 13:59:01.017774 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:01.017742    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8589c48f5d-9ldlx_56cec59c-ac3b-4480-aed5-2ec531d5ed27/kube-auth-proxy/0.log"
Sep 11 13:59:01.101642 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:01.101599    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k7gg4_c2f407a1-0426-4f25-bea0-9aa10bbc6e14/payload-pre-processing/0.log"
Sep 11 13:59:01.136701 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:01.136672    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"
Sep 11 13:59:02.144807 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:02.144776    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-9f4s6_e58cd463-a541-401b-99d7-9f976a8e6222/serve-healthcheck-canary/0.log"
Sep 11 13:59:02.823337 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:02.823292    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvnhq_a7d7905c-8a44-41c9-9e66-9ce00ac29496/kube-rbac-proxy/0.log"
Sep 11 13:59:02.876733 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:02.876697    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvnhq_a7d7905c-8a44-41c9-9e66-9ce00ac29496/exporter/0.log"
Sep 11 13:59:02.902526 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:02.902501    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-insights_insights-runtime-extractor-hvnhq_a7d7905c-8a44-41c9-9e66-9ce00ac29496/extractor/0.log"
Sep 11 13:59:03.250911 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:03.250885    2551 kubelet.go:2654] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-9fc22/perf-node-gather-daemonset-gdx2w"
Sep 11 13:59:06.485072 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:06.485036    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_llmisvc-controller-manager-777c6bd7d5-gkkmt_cd6ef683-4bf0-4601-97df-f5760a05cf62/manager/0.log"
Sep 11 13:59:06.563482 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:06.563459    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_maas-controller-58558d56b9-n4skp_d3019472-539f-428c-8a39-3f76e09cab76/manager/0.log"
Sep 11 13:59:06.639955 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:06.639929    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_model-serving-api-54d89bd5fc-zdqvf_b496961b-a099-47ec-8a34-de2690f7aefd/server/0.log"
Sep 11 13:59:07.056041 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.056012    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64959b9bbc-flljd_56a7e15d-c3f0-4460-be97-ee7dc88886a9/manager/0.log"
Sep 11 13:59:07.121514 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.121440    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/opendatahub_opendatahub-operator-controller-manager-64959b9bbc-flljd_56a7e15d-c3f0-4460-be97-ee7dc88886a9/copy-manifests/0.log"
Sep 11 13:59:07.592715 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.592687    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dbrjc_66485700-96fb-43e7-b7ce-48c63f234d47/discovery/0.log"
Sep 11 13:59:07.628003 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.627978    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8589c48f5d-9ldlx_56cec59c-ac3b-4480-aed5-2ec531d5ed27/kube-auth-proxy/0.log"
Sep 11 13:59:07.745122 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.745091    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k7gg4_c2f407a1-0426-4f25-bea0-9aa10bbc6e14/payload-pre-processing/0.log"
Sep 11 13:59:07.880196 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:07.880108    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"
Sep 11 13:59:09.822875 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:09.822840    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_istiod-openshift-gateway-5677ccb49b-dbrjc_66485700-96fb-43e7-b7ce-48c63f234d47/discovery/0.log"
Sep 11 13:59:09.900662 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:09.900622    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-8589c48f5d-9ldlx_56cec59c-ac3b-4480-aed5-2ec531d5ed27/kube-auth-proxy/0.log"
Sep 11 13:59:09.977533 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:09.977505    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-pre-processing-59fbc47bf8-k7gg4_c2f407a1-0426-4f25-bea0-9aa10bbc6e14/payload-pre-processing/0.log"
Sep 11 13:59:10.023943 ip-10-0-141-50 kubenswrapper[2551]: I0911 13:59:10.023910    2551 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_payload-processing-5b54b7fbb7-k65dc_c2ebb4a2-0943-4a41-bfbc-449854d04c77/payload-processing/0.log"